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ABSTRACT 


1.  The  purpose  of  this  project  was  to  construct  a new  series  of  examination 
materials  for  measuring  the  end-of -course  achievement  of  students  in  naval 
officer  candidate  programs.  The  examinations  were  to  test  "integrated  knowledge" 
rather  than  mere  memory  for  factual  information.  For  the  purposes  of  the  project, 
an  examination  was  defined  a a measuring  "integrated  knowledge"  if  it  incorporated 
a substantial  proportion  of  items  measuring  problem-solving  and  decision-making 
ability.  These  items  would  presumably  draw  upon  a variety  of  knowledges  and  at 
the  same  time  test  the  ability  to  apply  these  knowledges  in  realistic  situations. 

2.  On  the  basis  of  the  above  definition  of  an  "integrated  examination," 
a new  set  of  examinations  in  naval  science,  to  be  administered  as  year-end 
achievement  tests  to  NROTC  students  in  May  1953,  was  prepared.  These  examina- 
tions, while  incorporating  some  materials  which  had  been  used  in  the  examinations 
of  previous  years,  included  a substantial  proportion  of  new  items  judged  to 
measure  problem-solving  and  decision-making  ability.  In  the  course  of  preparing 
these  examinations,  the  examinations  in  use  as  final  achievement  tests  at  the 
Officer  Candidate  School,  Newport,  R.  I.,  were  extensively  revised.  All  examina- 
tions were  constructed  to  parallel  as  closely  as  possible  the  respective  curri- 
cula of  the  NROTC  and  OCS  programs. 

3.  In  statistical  analyses  designed  to  evaluate  the  examinations  con- 
structed, the  following  conclusions  concerning  the  OCS  tests  were  reached: 

(a)  The  time-limits  which  had  been  set  were  appropriate  for  measuring 
power  rather  than  speed. 

(b)  The  tests  met  reasonably  satisfactory  standards  of  reliability  and 
score -dispersion.  Further  improvement  in  test  reliability  would, 
however,  be  desirable. 
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(c)  The  subject-matter  covered,  by  the  tests  was  in  general  very 
heterogeneous. 

(d)  1 The  scores  for  the  various  department  examinations  were  highly  cor- 

related, suggesting  a single  general  factor  of  achievement  in 
Officer  Candidate  School. 

(e)  The  items  of  the  tests  showed  a reasonably  satisfactory  distribution 
of  item  difficulty; • 

(f)  A reasonably  large  number  of  items  showed  satisfactory  validity 

; against  a criterion  which  consisted  of  t he  total  weighted  scores  on 

; . ' all  examinations* 

(g)  High  validities  were  found  equally  often  for  items  appearing  to 
measure  sheer  factual  Information  and  for  items  appearing  to  require 
problem-solving  and  decision-making. 

It  should  be  pointed  out  that  no  truly  satisfactory  criterion  (based,  for  example, 
upon  observations  of  shipboard  performance)  was  available  for  evaluating  the  new 
examination  materials  which  were  constructed. 

ii.  An  effort  was  made  to  sort  out  the  items  of  the  OCS  examinations  in  terms 
of  different  varieties  of  achievement.  The  Wherry-Gaylord  iterative  factor- 
analysis.  procedure,  applied  to  72  representative  items,  yielded  five  linearly  inde- 
pendent factors.  This  analysis  was  later  expanded  to  include  all  ulQ  items  of  the 
examinations.  One  factor  was  interpreted  as  a residual  error  factor.  The  four 
factors  which  remained  were  rather  highly  correlated,  a result  which  suggested 
that  their  isolation  and  separation  was  chiefly  due  to  the  capitalization  of  chance 
independencies.  There  was  certainly  no  strong  evidence  for  the  separation  of  dif- 
ferent varieties  of  achievement  in  OCS,  On  the  contrary,  the  chief  conclusion 
was  that  achievement  in  OCS  is  quite  uniform  j that  is,  individuals  tend  to  perform 
at  the  same  level  of  achievement  in  all  aspects  of  t he  curriculum.  Furthermore, 
individuals  tend  to  perform  at  the  same  level  of  achievement  regardless  of  whether 
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they  are  tested  with  items  covering  factual  information  or  with  items  appearing 
to  measure  problem-solving  arid  decision-making  ability.  Nevertheless,  to  the 
extent  that  the  factors  identified  by  the  Wherry-Gaylord  procedure  could  be  in- 
terpreted, the  following  types  of  achievement  seem  to  exist  with  at  least  some 
degree  of  independence*. 

I.  Verbal  reasoning  and  judgment  (Key  E100) . 

II.  Spatio-temporal  reasoning  (Keys  E200  and  E300,  which  were  quite 
highly  correlated). 

III.  Ability. to  remember  factual  information  (Key  EiiOO). 

5.  '.aa  technique  of  "individual  operating  characteristics"  was  applied  to 
the  interpretation  of  the  factor  keys.  This  technique  is  designed  to  present 
the  probability  with  which  an  individual  getting  any  given  score  will  pass  items 

of  any  given  difficulty.  The  technique  thus  enables  the  interpretation  of  test 

« ‘ 

scores  in  terms  of  actual  performance  on  items  rather  than  merely  in  terms  of 
norms.  Complete  data  on  the  operating  characteristics  of  OCS  factor  scores  are 
presented  in  the  body  of  this  report.  The  graphical  presentations  of  operating 
characteristic  curves  confirm  the  observation  made  elsewhere  that  the  content 
of  the  examinations  is  extremely  heterogeneous. 

An  attempt  was  made  to  compare  groups  of  OCS  and  NROTC  students  with 
respect  to  performance  on  naval  science  achievement  tests.  When  representative 
groups  of  OCS  and  NROTC  students  were  compared  on  material  common  to  the  two 
curricula  and  freshly  reviewed  by  both  groups,  the  NROTC  students  were  on  tne 
avera^  superior  to  the  OCS  students.  This  was  particularly  true  of  3rd  ana  i+th 
year  NROTC  studentsj  this  result  suggests  the  possible  operation  of  a selective 
factor  throughout  the  four-year  NROTC  curriculum. 

However,  when  OCS  students  were  compared  with  Uth  year  NROTC  students 
on  material  covered  in  the  first  three  years  of  the  NROTC  course  and  hence  not 
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freshly  reviewed  by  the  NROTC  students,  the  OCS  students  were  clearly  superior, 

| particularly  so  after  appropriate  statistical  controls  were  imposed.  Thus,  even 

1 though  Uth  year  NROTC  students  are  superior  to  OCS  students  on  material  covered  in 

the  lpth-year  NROTC  curriculum,  they  show  considerable  lack  of  retention  for  the 
j material  covered  in  earlier  years. 

RECOMMENDATIONS  FOR  FURTHER  RESEARCH 

1.  In  the  construction  of  new  examination  materials,  the  approach  utilized  in 
i the  present  project  should  be  further  explored.  Items  identified  in  this  report  as 

appearing  to  measure  problem-solving  and  decision-making  ability  might  be  used  as 
k models  for  the  construction  of  new  examination  items.  Also,  this  approach  could  be 

extended  into  certain  areas  of  the  naval  science  curriculum  which  were  not  thor- 

l 

oughly  worked  on  in  the  present  project,  such  as  Engineering  and  Damage  Control,  and 

| 

Military  Justice.  Descriptions  of  realistic  situations  which  would  provide  the 

necessary  context  for  the  construction  of  items  measuring  problem-solving  and  de- 

i 

cision-making  abilities  might  be  developed. 

2.  It  would  be  desirable  to  validate  the  special  factor  keys  developed  in  the 
present  research  against  on-the-job  shipboard  performance.  They  may  also  be  used 
as  tentative  criteria  for  new  items  which  may  be  consti'ucted,  and  for  this  purpose 
the  items  composing  these  factor  keys  should  continue  to  be  included  in  future 
examinations.  At  the  same  time,  further  research  could  be  undertaken  to  examine 
the  factorial  composition  of  naval  science  examinations  in  order  to  see  whether 
further  evidence  can  be  found  for  differentiating  factual -knowledge  examinations 
from  what  have  been  called  "integrated"  examinations  in  the  present  research. 
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FOREWORD 
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INTRODUCTION 


This. project  was  initiated  in  an  attempt  to  provide  an  improved  examination 
4 program  to  be  used  by  the  Navy  in  evaluating  the  achievement  of  students  enrolled 

in  schools  for  officer  candidates. 

Before  the  inception  of  the  project,  the  Bureau  of  Naval  Personnel  had 
expressed  a desire  to  improve  the  examinations  then  in  use.  The  Bureau  felt  that 
a new  series  of  examinations  designed  to  test  the  candidate’s  "integrated  know- 
ledge" of  the  subject-matter  of  Naval  Science  courses  would  provide  a more  valid 
means  of  assessing  performance  and  of  assigning  lineal  precedence  to  graduates, 
as  required  by  law. 

In  addition,  the  Bureau  ,cf  Naval  Personnel  had  expressed  a desire  to  obtain 
a means  of  establishing  objective  comparisons  among  trainees  in  the  three  types 
of.  general  line  programs,  the  Officer  Candidate  School  (OCS),  the  Naval  Reserve 
Officer  Training  Programs  (NROTC),  and  the  Reserve  Officer  Candidate  Program 
* ;•  (ROG).  These  programs  differ  considerably  with  espect  to  purpose,  type,  and 

duration  of  training,  and  recruiting  of  trainees: 

(a)  The  Officer  Candidate  School  at  the  U.S.  Naval  Base,  Newport,  R.  I.  conducts 
an  18-week  course  emphasizing  "practical,”  vocational  aspects  of  all  phases 
of  basic  naval  science.  Students  are  college  graduates. 

(b)  The  Naval  Reserve  Officer  Training  Program  is  conducted  at  52  colleges  and 
; universities,  in  conjunction  with  the  regular  academic  courses  at  these 

institutions.  The  course  lasts  four  years,  and  covers  both  "practical" 
and  "theoretical”"  aspects  of  naval  science,  with  an  emphasis  on  the  latter. 

. (c)  The  Reserve  Officer  Candidate  Program  is  conducted  at  one  or  more  naval 

installations  during  the  summer.  The  trainees  are  college  students  and 

4 

others  who,  since  they  are  not  attending  colleges  having  NROTC  Un^ts,  find 
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it  possible  to  take  this  training  only  daring  the  simmer.  The  curriculum 
is  similar  to  that  established  for  OCS,  but  may  depart,  from  it  in  sene  respects, 
if  only  because  of  the  fact  that  the  program  is  conducted  at  naval  installations 
other  than  Newport  and  hence  with  a different  instructional  staff.  » 

In  view  of  the  differences  among  these  programs*  the  Bureau  of  Naval  Personnel 
has  not  felt  it  wise  to  prescribe  identical  final  Examinations  for  them.  Hence, 
the  problem  of  comparing  these  programs  in  some  objective  manner  has  appeared 
resistant  to  any  simple  solution. 

In  order  to  solve  the  problems  posed  by  the  above-cited  needs  of  the  Navy, 
it  was  believed  that  certain  new  procedures  of  test  analysis  would  have  to  be 
developed  and  applied. 

Accordingly,  this  project  established  three  basic  objectives: 

1.  The  modification  of  existing  final  achievement  examinations  for  naval  officer 
candidates  to  meet  more  adequately  the  requirements  set  forth  by  the  research 
sponsor,  namely,  that  the  examinations . should  measure  the  extent  to  which 
candidates  could  demonstrate  an  n integrated*  knowledge  of  naval  science  subjects 
rather  than  merely  memory  for  factual  details.  The  examinations  would  be 
designed  to  measure  the  ability  to  solve  specific  problems  requiring  technical 
and  professional  skill,  and  the  ability  to  make  sound  decisions  in  novel 
situations. 

2.  The  development  and  application  of  a technique  for  comparing,  with  respect 

to  performance  at  the  end  of  the  course,  naval  officer  candidates  enrolled 
in  the  three  general  line  programs  described  above:  NROTC,  OCS,  and  ROC. 

3.  The  accomplishment,  so  far  as  possible,  of  basic  research  into  methods  of 
investigating  the  content  homogeneity  of  achievement  examinations  and  methods 
of  scaling  and  interpreting  test  performances 


i 
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it  the  outset,  the  work  of  the  project  was  planned  in  two  phases.  Phase  I 
involved  the  construction  of  a series  of  comprehensive  examinations  in  naval 
science,  and  thus  was  designed  to  fulfill  research  objective  (1)  above.  Phase  II 
was  to  be  concerned  with  the  detailed  analysis  of  data,  and  hence  with  research 
objectives  (2)  and  (3).  Phase  II  was  to  be  undertaken  only  after  Phase  I was 
nearly  complete. 

This  report  is  divided  into  two  parts,  corresponding  to  these  phases  of 

work. 
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PART  ONE 


' ■ THE  CONSTRUCTION  OF  ACHIEVEMENT  EXAMINATIONS  IN  NAVAL  SCIENCE 
CHAPTER  I.  DEFINING  EXAMINATION  OBJECTIVES 


The  Examination  and  the  Curriculum 

n,  , . the  function  of  ROTC  is  to  implant  in  a prospective  officer 
the  knowledge  of  his  craft  which  he  must  have  if  he  is  to  function 
ae  an-  officer  when  he  goes  on  active  -duty.  It  reminds  us  that  any 
specialized  profession  calls  for  memorizing  a tody  of  exact  facts  — 
medical  students,  for  example,  have  to  learn  anatomy,  and  they  are 
apt  not  to  like  it  in  contrast  to  the  more  speculative  subjects  they 
studied  in  college.  How  well  a young  officer  has  mastered  the  details 
of  his  training  may  spell  the  difference  between  survival  and  death  for 
himself  and  the  men  under  his  command.  An  infantry  officer  must  be 
able  to  work  a gunj  whether  he  understands  the  chemistry  of  explosives 
is  purely  incidental  to  his  responsibilities  as  a platoon  leader  or 
company  commander.  To  know  what  buttons  to  push  on  an  automatic  cal- 
culator is  more  important  to  a young  antiaircraft  officer  than  to 
comprehend  the  mathematical  theories  that  made  it  possible.  Admittedly 
this  sort  of  knowledge  or  skill  is  not  a normal  element  of  a college 
education^  but  this  is  not  to  say  tnat  as  a field  of  study  military 
science  must  live  in  a narrow  isolation  from  the  academic  curriculum. 

Total  war  is  more  than  a strictly  military  problem.  The  'know  why' 
is  an  essential  element  of  the  ’know  how',  and  should  be  part  of 
the  equipment  of  an  ROTC  graudate.” 

So  writes  President  Dodds  of  Princeton,  in  presenting  the  case  for  professional 
military  education  in  civilian  colleges  and  universities,  in  a recent  article  in 
the  Atlantic  Monthly  (March,  19!?3)« 

The  conflicting  demands  of  fact  and  theory  in  professional  education  have 
never  been  adjudicated  to  the  complete  satisfaction  of  everyone,  and  probably 
never  will  be.  On  the  one  hand,  there  always  exist,  in  any  field  of  professional 
training,  more  than  enough  facts  to  fill  a curriculum  to  bursting,  and  for  each 
of  these  facts  one  can  find  experts  who  will  say  that  its  acquisition  is  an  abso- 
lute necessity  on  the  part  of  the  trainee.  On  the  other  hand,  there  are  those 
who  will  say  that  a deep  knowledge  of  theory  will  suffice  to  get  any  man  by  in 
any  situation. 


In  the  formation  of  a curriculum,  the  conflict  between  fact  and  theory  can 
often  be  bypassed,  on  the  ground  that  the  conflict  can  be  compromised  by  teaching 
fact  and  theory  in  equal  measure,  but  in  the  design  of  an  examination  to  cover 
this  curriculum  the  conflict  must  be  faced  openly.  An  examination  is  a device 
for  evaluating  the  achievement  of  the  student  in  terms  of  the  behaviors  he  is 
expected  to  have  acquired  as  a result  of  training.  What  kind  of  behavior  is 
desired  in  a naval  officer  candidate  at  the  end  of  his  training?  This  question, 
in  the  broad  setting  we  have  Just  described,  is  one  which  had  to  be  faced  in  the 
conduct  of  the  present  project. 

The  sponsor  of  the  project,  the  Bureau  of  Naval  Personnel  specified  that  the 
examinations  which  were  to  be  constructed  for  the  evaluation  of  Naval  Officer 
Candidate  graduates  should  be  truly  "comprehensive  and  integrated,"  not  in  the 
sense  of  a rigorous  coverage  of  the  subject-matter,  but  in  terms  of  a better 
sampling  of  the  actual  skills  and  knowledges  demanded  of  the  NROTC  graduates 
when  assigned  to  the  fleet.  It  suggested  that  the  tasks  set  in  the  examinations 
should  be  concerned  with  the  ability  to  make  decisions  in  novel  situations  and 
the  ability  to  solve  problems,  and  that  at  the  same  time  the  solution  of  these 
tasks  should  call  for  a wide  range  of  factual  knowledge,  so  that  the  tests  would 
measure  the  extent  to  which  examinees  had  integrated  their  knowledge  in  the  whole 
context  of  tie  material  presented  in  the  curriculum.  It  is  thus  seen  that  the 
Bureau  of  Naval  Personnel  took  a middle  position  with  respect  to  the  relative 
roles  of  theory,  fact,  and  practical  application. 

The  project  staff  sought  to  arrive  at  a more  specific,  operationally 
applicable  definition  of  "comprehensiveness"  and  "integration"  with  reference 
to  the  actual  conditions  surrounding  the  project.  The  conditions  of  which  we 
speak  had  to  do  with  the  facilities  available  for  tryout  of  the  examinations 
and  the  present  state  of  instructional  programs  where  testing  was  to  be  conducted. 


- 5 - 


Specifically,  it  was  3een  necessary  to  suit  the  examinations  to  actual,  current 
curricula  rather  than  to  the  more  nearly  ideal  curricula  which  might  be  estab- 
lished at  some  time  in  the  future*  Furthermore,  since  the  examination  materials 
which  were  eventually  to  be  used  in  connection  with  the  NROTC  program  had  to  be 
developed  in  the  context  of  instruction  at  the  Officer  Candidate  School  at  Newport, 
Hi  I*,  a continual  view  of  what  was  practicable  in  the  light  of  differences  between 
these  curricula  had  to  be  maintained* 

■ Thrde  ways  of  defining  the  concept  of  a "comprehensive,  integrated  examination" 
in  naval  science  were  considered,  and  although  all  were  in  some  measure  useful  in 
ghlding  efforts  in  test  construction,  only  one  of  them  was  found  to  be  realistic 
as  a primary  point  of  departure.  The  three  definitions  are  listed  below,  with 
comments  on  their  relevance  to  possible  kinds  of  items  in  the  situation  outlined 
above  * 

1.  "Integrated"  items  could  be  conceived  as  those  which  test  organized 
combination^  of  subject-matter  from  different  fields.  For  example,  one  could 
set  a problem  which  involved  facts  and  principles  from  two  fields,  such  as 
Weapons  and  Operations  . 

This  definition  was  rejected  because  the  NROTC  curriculum  is  organized  in 
such’  a manner  that  subject  fields  are  taught  over  a period,  of  four  years.  It 
would  therefore  be  impossible  to  construct  a truly  comprehensive  test  covering 
several  subject  fields  for  each  year  of  Naval  science.  An  alternative  would  be 
to  construct  cumulative  achievement  examinations  for  administration  at  the  end  of 
each  year,  but  such  tests  would  be  essentially  tests  of  retention  and  would  require 
NROTC  units  to  provide  cumulative  reviews.  However,  this  definition  might  have  been 
workable  if  tests  were  being  prepared  solely  for  the  Officer  Candidate  School  where 
all  the  subject  fields  are  taught  concurrently, 

t . : 

The  OCS  curriculum  is  organized  in  six  departments!  Orientation  and  Military 
Justice,  Naval  Weapons,  Operations,  Seamanship,  Navigation,  and  Engineering* 
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2.  "Integrated"  items  could  be  conceived  as  those  which  test  combinations 
of  knowledges  required  in  the  duties  performed  as  an  Ensign.  Thus,  the  items 
would  in  effect  constitute  "verbalized  performance"  items.  Each  item  would  aak 
the  examinee  to  state  how  he  would  perform  in  a certain  situation  and  it  would 
require  an  integrated  knowledge  of  several  facts  in  order  to  arrive  at  a correct 
statement  of  proper  performance, 

. This  definition  was  rejected  in  principle  for  two  reasons s 
a*.  If  valid  items  were  to  be  constructed  on  the  basis  of  combinations 

cf  knowledges  required  of  a newly  commissioned  Ensign  in  the  performance 
of  his  duties  aboard  ship,  detailed  job  analyses  of  these  duties  and 
requirements  would  have  to  be  available.  It  must  be  remembered  that 
the  duties  and  responsibilities  of  Ensigns  differ  greatly  depending 
upon  the  types  of  vessel  to  which  they  are  assigned.  Thus  the  re- 
sponsibilities of  an  Ensign  on  a battleship  are  not  identical  to  those 
of  an  Ensign  aboard  an  LST,  Therefore,  several  job  analyses  would  be 
required,  corresponding  to  the  duties  of  Ensigns  aboard  different  types 
of  naval  craft.  Such  job  analyses  were  not  available  to  project 
personnel.; 

b.  , Even  if  the  requisite  types  of  job  analyses  were  available,  their  use 
, in  the  construction  of  achievement  examinations  for  officer  candidates 

would  be  questionable  in  the  light  of  present  NROTC  and  OCS  curricula. 
Training  in  the  officer  candidate  programs  (line)  is  not  specific  to 
certain  ships.  The  curriculum  intent  is  to  acquaint  the  trainees  with 
the  Navy  and  with  naval  procedures  and  to  provide  them  with  a broad 
background  of  knowledges  and  requirements  connected  with  the  operation, 
maintenance,  and  armament  of  naval  vessels  in  general.  Nevertheless, 
newly -comissioned  Ensigns  graduating  from  OCS  are  not  expected  to  be 
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able  to  perform  shipboard  duties  except  at  the  lowest  leTOl  of  'responsibility. 
It  is  only  after  they  receive  several  months  of  shipboard,  on-the-job' 
training  in  specific  duties,  that  they  are  entrusted  with  independent  re- 
sponsibility* Thus,  examinations'  based  -on  the  duties  of-  fleet  Sasigns 
might  be  in  conflict  with  the  intent  and  design1 of  officer  candidate 
program  curricula*  : V ; r 

3*  Lastly,  “integrated"  items  might  be  conceived  as  those  which  incorporate 
decision-making  or  problem-solving  functions  by  the  integration  of1  knowledge-} 
i.e.,  rather  than  testing  on  a specific  knowledge  "A,"  each  item  would  require  the 
examinee  to  manipulate  knowledges  "A,"  ?By"  and  "C"  in  order  to  arrive  at  the  correct 
answer  "D.n  In  addition,  the  item  would  require  the  examinee  to  perceive  relations 
and  implications  existing  between  various  faots.  These  "knowledges"  represent  items 
of  instruction  to  which  the  candidate  has  been  exposed  during  the  Course  of  the 
officer  candidate  training  program.  . . ' 

This  last  definition  was  accepted  to  guide  the  work  of  construction  in  the 
test  items.  ’ • 

It  was  assumed  that  items  conforming  to  definition  3,  and  also- in  some  degree 
to  definition  2,  could  best  be  constructed  by  outlining  typical • sequences  of 
activities  in  which  a new  Ensign  might  be  engaged,  and  building  items  based  upon 
these  reaslistic  situations.  Such  situations  usually  involve  a complex  Of  factors 
to  be  taken  into  consideration  in  solving  problems.  Hence,  the  examinations  are 
expected  to  be  predictive  of  how  well  examinees  will  be  able  to  solve  problems  arising 
in  their  shipboard  on-the-job  training  and  in. their  later  naval  careers. 

The  extent  to  which  it  ia  possible  to  test  "integrated"  knowledge  is  partly 
a function  of  the  form  of  the  examination.  The  above  considerations  relating  to 
ways  of  measuring  "integrated  knowledge"  apply  largely  to : examinations  of  the 
multiple -choice  type.  At  a conference  of  project1  consultants  held -on  lii  December  19!?2, 
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various  alternative  means  of examining  were  proposed.  For  example.  Dr.  Noi'man 
Frederiksen  suggested  that  examinees  be  shown  a sound  film  on  which  they  would 
take  notes.  This . sound  film  would  present  shipboard  situations  in  a realistic 
fashion.  At  appropriate  points,  the  examinee  would  be  asked  to  make  certain 
decisions.  After  the  correct  answer  was  given,  the  film  would  proceed  to  other 
situations,  to  be  handled  in  a similar  fashion.  Dr.  Frederiksen  also  pointed 
out  that  the  multiple-choice  format  reduces  the  opportunity  for  the  individual 
to  demonstrate  creative  problem-solving  ability.  It  would  perhaps  be  feasible, 
he  suggested,  to  have  examinees  write  solutions  to  problems  in  essay  form,  in 
duplicate.  To  adapt  this  procedure  for  objective  scoring,  one  would  have  the 
examinee  turn  in  one  copy  of  his  essay  answer,  then  code  the  copy  he  retains  in 
terms  of  multiple-choice,  options.  Finally,  both  Dr.  Frederiksen  and  Dr.  Rulon 
suggested  that  work-sample  or  performance  tests  be  constructed  as  a further  means 
of  testing  "integrated  knowledge."  . Unfortunately,  due  to  project  deadlines 
(which  called  for  the  construction  of  the  NROTC  examinations  by  15>  February  1953), 
none  of  these  suggestions  could  be  followed  in  the  actual  work  of  the  project. 

It  should  be  emphasized  that  the  NAVPERS  Officer  Candidate  Achievement 
Examinations  are  not  used  as  a basis  for  disqualification.  When  necessary, 
trainees  are. either  disenrolled  or  "washed  back"  prior  to  the  stage  at  which 
the  achievement  examinations  are  administered.  Grades  on  these  examinations  do, 
however,  comprise  one  of  several  criteria  employed  ty  the  Navy  to  establish  the 
lineal  precedence  of  commissioned  officers  within  their  respective  jchool 
classes.  Thus,  there  seems  some  basis  for  the  belief  that  examinees  are  fairly 
well  motivated.  Project  personnel  gained  the  impression  that  motivation  in  taking 
the  examinations  is  especially  high  at  NROTC  units. 


CHAPTER  II.  TEST  CONSTRUCTION  PROCEDURES 


At  the  Officer  Candidate  School  (Navy),  the  curriculum  covers  six  areas  of 
instruction:  Orientation  and  Military  Justice , Operations,  Seamanship,  Navigation, 

Naval  Weapons , and  Engineering  and  Damage  Control.  Instruction  at  'OCS  is  given 
concurrently  in  the  six  areas  during  the  15  week  course.  ' " ' 

The  NROTC  program  consists  of  training  given  during  four  college  years  in 
certain  areas  as  follows:* 

(Freshmen)  Naval  Science  I - Orientation,  Seamanship,  Operations 
(Sophomores)  Naval  Science  II  - Naval  Weapons,  Operations 
(Juniors)  Naval  Science  III  - Navigation,  Seamanship,  Operations 
(Seniors)  Naval  Science  IV  - Military  Justice,  Engineering,  Seamanship. 

It  might  appear  from  the  above  that  the  NROTC"  and  OCS  curricula  are  practically 
identical.  .Appearances,  however,  are  deceptive.  Although  overlap  existed-  between 
the  two  curricula  in  some  areas  of  instruction,  there  were  large  differences  in 
others.  In  effect,  examinations  had  to  be  separately  planned  and  constructed  for 
use  in  OCS  and  in  NROTC. 

At  the  outset,  project  personnel  were  provided  with  copies  of  NAVPERS  achieve- 
ment examinations  administered  to  NROTC  midshipmen  at  the  end  of  each  of  the  four 
years  of  training.  These  sets  of  examinations  were-  dated  May  1951  and  May  1952, 
NAVPERS  achievement  examinations  administered  in  May  1952  in  the  six  OCS  areas  of 

instruction  were  also  provided.  Copies  of  quarterly  examinations  and  quizzes 

. 

This  breakdown  with  respect  to  areas-  of  instruction  at  NROTC  is  an  impressionistic 
summary  of  the  curriculum  as  prescribed  every  year  in  an  official  BuPe  publication 
(e.g. , U.S.  Navy  Standard  Curriculum  for  NROTC  (line),  NAVPERS  91828).  The  NROTC 
program  is  categorized  by  the  navy  simply  in  terms  of  years,  e.g.  N8-1,  Nfl-2, 
rather  than  by  subject  material* 


administered  to  trainees  at  the  Officer  Candidate  School  were  also  made  avail- 
able. Finally,  item  analyses  of  the  1951  and  1952  NROTC  NAVPEE3  achievement  exam- 
ination? were  furnished  to  aid  in  the  item  construction  task. 

A.  Construction  of  Achievement  Examinations  for  Officer  Candidates  (PCS) 

The  initial  task  of  the  project  was  to  revise  the  OCS  May  1952  NAVPER5 
achievement  examinations  in  the  six  subject  areas  for  administration  to  officer 
candidates  at  Newport  just  prior  to  graduation. 

First,  the  1952  OCS 'eicaminations  and  the  May,  1952  NROTC  examinations  were 
subjected  to  detailed  inspection  for  two  purposes:  (a)  to  gain  an  impression 

of  the  types  of  items  previously  in  use,  and  (b)  to. evaluate  the  subject-matter 
coverage  of  existing  examinations.  This  inspection  of  previous  examinations  was 
carried  out  by  project  personnel  in  collaboration  with  appropriate  instructional 
personnel  at  the  Newport  OCS.  For  the  purpose  of  appraising  item -types,  items 
were  classified  in  three  categories:  "problem-solving,"  "factual -informational," 

and  "decision-making."  An  attempt  was  made  to  introspect  into  the  mental  processes 
which  examinees  would  have  to  employ  in  order  to  answer  the  items  correctly. 
Consideration  was  given  to  the  nature  of  the  instruction  which  had  been  received} 
for  example,  an  item  which  might  appear  to  an  outsider  a "problem-solving"  item 
might  be  judged  merely  "factual-informational"  because  the  item  called  for  a 
set  solution  presented  in  the  classroom  as  something  to  be  memorized.  While 
it  is  recognized  that  mental  functions  overlap,  an  attempt  was  made  to  classify 
items  in  terms  of  the  chief  or  crucial  element  involved. 

As  was  expected,  the  large  majority  of  items  on  all  examinations  were  of 
the  informational  type,  as  may  be  seen  from  Table  2.1.  The  single  exception 
was  in  the  navigation  examinations,  which  contained  actual  navigational  problems; 
hence  the  items  were  classified  as  "problem-3olving."  It  was  seen,  in  any  case, 
that  a guiding  principle  in  revising  the  tests  wo:  .Id  be  an  attempt  to  provide 
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more  items  of  the  "problem-solving"  and "decision-making"  types*  Nevertheless, 
it  was  recognized  that' revised  examinations  should  contain  a reasonable  proportion 
of  "factual-informational"  items,  • 


TABLE  2,1 

himhsrs  of  items  classified  in  each  of  three  categories 

May  1952  OCS  and  NROTC  Eixaminations 


• FACTUAL^ 

INFORMATIONAL 

PROBLEM- 

SOLVING 

DECISION- 

MAKING 

TOTAL 

i 

> 

* 

No. 

% 

No. 

% 

No. 

% 

No. 

, 

OCS 

Orientation  and 
Military  Justice 

62 

82.7  ' ■ 

0 • 

O' 

13 

17.3 

75 

OCS 

Operations 

55 

73.3 1 * 

' 17  ' 

22.7', 

3 

k.o 

75 

| 

■ 

OCS 

Seamanship 

32  * 

71.1 

3 

6.7 

10 

22.2 

U5 

OCS 

Navigation 

2U 

liO.O 

36 

'60.0 

0 

0 

60 

OCS 

Naval  Weapons 

71 

9k.  7 

k ' 

5.3 

0 

0 

75 

OCS 

Engineering  and 
Damage  Control 

71 

9k.7 

. 4 

5.3 

0 

0 

75 

V 

' 

NROTC 

First  Tear 

11*5 

96.7 

2 

1.3 

3 

2.0 

150 

] 

NROTC 

Second  Tear 

137 

91.3 

11  , 

,7.3. 

2 

1.3 

150 

' 

NROTC 

‘Third  Tear  Part  I 

55 

73.3  . 

13 

17.3 

7 

9.k 

75 

' 

NROTC 

Third  Tear  Part  II 

0 

0 

30 

100.0 

0 

0 

30 

NROTC 

Fourth  Tear 

13k 

89.3 

5 

3.3 

11 

7.3 

150 

With  respect  to  coverage  of  the  curriculum,  it  appeared  that  the  existing 
examinations  represented  a balanced  sampling  of  material}  that'  is,  they  emphasized 
various  portions  of  the  curriculum  roughly  in  proportion  to  the  time  spent  in 
teaching  these  portions. 
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Upon  completion  of  this  analysis  of  existing  examinations)  test  item  fi3es 
were  established  for  each  of  the  six  areas  of  OCS  instruction. 

Next,  actual  construction  of  test  items  was  undertaken  for  the  final  exam- 
inations to  be  administered  to  OCS  Class  VII  in  October  1952.  The  limited 
objective  established  at  that  time  was  to  modify  existing  NAVPERS  OCS  examinations 
so  that  (a)  the  content  would  be  more  closely  aligned  with  the  curriculum  currently 
in  use  or,  with  planned  changes  in  the  curriculum  and  (b)  only  items  regarded  as 
useful  in  measuring  overall  competence  of  the  officer  candidate  would  be  retained. 
Although  many  factual  items  would  be  retained,  effort  was  to  be  expended  in  selecting 
or  constructing  items  measuring  a more  integrated  type  of  knowledge  with  the 
emphasis  on  problem  solving,  situational  items. 

Test  items  in  the  six  examinations  were  editorially  improved,  completely 
revised  or  rewritten  according  to  the  following  procedures 

1.  Based  upon  the  curriculum  and  study  guides,  a new  tentative  test  outline 
was  developed  for  each  of  the  six  OCS  examinations. 

2.  For  each  examination  the  backlog  of  items  was  assembled  in  accordance 
with  the  new  test  outline.  In  those  cases  where  a number  of  items  existed  for 
a specific  outline  topic,  the  items  which  appeared  "best"  in  terms  of  content 
importance  and  measurement  characteristics  were  selected  by  project  personnel. 

Item  revisions  were  suggested  directly  on  the  item  cards. 

3.  At  this  point,  the  new  test  outlines  and  the  selected  items  and  reviews 
were  discussed  with  OCS  instructor  personnel,  In  collaboration  with  these 
instructors,  each  item  selected  by  project  personnel,  together  with  the  suggested 
revisions  was  reviewed.  Further  revisions  were  made  where  necessary.  If  an  item 
was  found  to  be  totally  unsuitable,  the  instructors  were  asked  to  inspect  the 
remaining  items  in  the  file  for  a possible  substitution  or  modification.  If 

no  suitable  substitute  item  was  available,  a new  item  was  constructed.  New 
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items  were  also  constructed  to  cover  areas  of  the  test  outline  for  which  no  items 
previously  existed.  Seme  items  required  the  use  of  pictorial  material,  which  was 
prepared  at  Newport  by  Navy  artists, 

li«  The  final  examination  items  were  edited  and  assembled  for  typing  and 
reproduction.,  The  placements  of  correct  answers  within  the  five  alternatives 
were  randomized* 

5, . In  order  to  insure  accuracy,  OCS. officials  reviewed  the  final  reproduction 


copy..  , 


The  reproduction  copy  for  the  six  examinations  to-  be  administered  to  Class  VII 
was  delivered  to  the  Academic  Director,  OCS,  Newport,  R.I.  along  with  three  IBM 
score  keys  for  each  test. 

Subsequent  to  the  administration  of  the  examinations  to  OCS  Class  VII,  item 
analyses  of  the  six  examinations  were  performed,  using  Flanagan's  procedure  in- 
volving groups  of  9,  20,  i;2,  20,  and  9 per  cent.*  The  average  grade  achieved  on 
the  six  achievement  examinations  constituted  the  criterion  against  which  all 
items  were  validated.  This  uniform  criterion  was  used  for  all  items  for  the  fol- 
lowing reasons:  (1)  It  was  desired  to  select  items  for  use  in  future  examinations 

which  would  measure  "integrated  knowledge,"  insofar  as  possible.  The  average 
grade  on-  all . six  examinations  was  believed  to  represent  the  best  criterion 
available  for  this  "integrated  knowledge."  (2)  The  use  of  a uniform  criterion 
would  effect  considerable  savings  in  the  time  required  to  prepare  the  data  for 
analysis.  The  alternative  procedure  would  have  been  to  U3e  the  total  score  on 
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asuli  djumiliiaiiun  aa  the  criterion  against  which  items  in  that  examination  would 


be  evaluated.  Such  a procedure  would  have  tended  to  select  items  uniquely 
measuring  different  subject-matters  rather  than  to  select  items  measuring  an 


Flanagan,  J.C,  The  effectiveness  of  short  methods  for  calculating  correlation 
coefficients.  Psychol.  Bull. , 1952,  U9;  3^2 -3^8 . 


integrated  knowledge  .of  the  whole  course..  The  actual  results  of  item  analysis 
will  be  presented  and  discussed  in  a later  section  of  this  report. 

In  the  light  of  item  analysis  data,  the  examinations  in  Seamanship,  Weapons 

|V  ' • 

and  Navigation  were  further  revised  for  administration  to  OCS  Class  VIII, 

December  1952  graduation.  The  examinations  in  Orientation  and  Military  justice. 
Operations , and  Engineering  and  Damage  Control  constructed  for  OCS  Class  VII 
(October,.  1952)  were  again  administered  to  OCS  Class  VIII,  with  only  very  minor 
revisions* 

Of  the  six  new  examinations  thus  far  constructed,  the  Naval  Weapons  was  most 
completely  revised.  This  examination  was  now  considered  to  be  directly  aligned 
with  the  OCS  curriculum.  For  example,  in  former  examinations  several  questions 
were  devoted  to  atomic  theory,  principles,  and  weapons.  These  were  eliminated 
from  the  new  examination,  since  atomic  weapons  and  theory  were  no  longer  taught 
at  OCS,.  The  examination,  items  are  for  the  most  part  n integrated"  items;  many 
are  in  situational  contexts.  That  is,. they  do  not  test  on  specific  details,  but 
require. the  integration  of  knowledges  acquired  throughout  the  course.  Use  was 
made  of  the  device  of  presenting  a realistic  developing  situation  and  then  asking 
a number, of  questions  relating  to  the  situation. 

The  Navigation  examination  (part  2)  has  always  contained  a celestial  problem 
requiring  a plot  of  a "day's  work."  This  problem  represents  an  integration  of 
all  knowledges  and  procedures  required  with  respect  to  navigating  a vessel 
at  sea  from  one  point  to  another.  The  problem  set  for  the  OCS  Class  VIII  was 
revised  in  accordance  with  the  1952  Nautical  Almanac.  Prior  examinations  were 
based  on  the  1950  almanac,  . 

The  Seamanship  examination  for  Class  VIII  was  revised  to  include  more  items 
on  Rules  of  the  Road;  otherwise,  it  underwent  no  great  change. 

Again,  the  six  examinations  administered  to  OCS  Class  VIII  were  subjected 


to  an  item  analysis 


Subsequently,  the  Operations  examination  was  completely  revised  for  adminis- 
tration to  OCS  Class  IX  (March  1953  graduation).  With  the  revision  of  this  exam- 
ination, work  was  completed  on  test  construction  for  OCS. 

Since  the  construction  of  the  March  1953  OCS  Operations  examination 
represented  one  of  the  project’s  most  thoroughgoing  attempts  to  prepare  an 
’’integrated"  examination  of  the  desired  type,  details  on  it3  preparation  may  be  of 
interest. 

In  December,  1952,  the  project  director  spent  a full  day  discussing  plans  for 
this  examination  with  Operations  instructors  at  Newport.  He  asked  them  to  envisage 
a typical  tour  of  duty  which  a newly-commissioned  officer  might  take,  and  to 
specify  the  kinds  of  experiences  which  they  might  have  on  such  a tour  of  duty. 

After  considerable  probing,  the  investigator  elicited  a description  of  experiences 
which  a new  ensign  might  typically  have  on  board  a destroyer  as  it  moves  from  a 
Navy  yard,  through  a shakedown  cruise,  and  out  to  fleet  operations  in  a combat 
area.  For  example,  the  new  Ensign  is  first  assigned  as  Custodian  of  Registered 
Publications  with  collateral  duties  as  Chief  Watch  Officer,  The  subject-matter 
experts  then  helped  the  investigator  draft  a number  of  item-idea3  based  on  this 
situation.  Questions  were  drafted  relating  to  the  chain-of -command  structure  under 
which  the  ensign  would  operate,  the  problems  involved  in  establishing  a file  of 
registered  publications,  the  guarding  of  radio  channels,  the  preparation,  evalua- 
tion, and  distribution  of  messages,  etc.  In  a later  phase  of  his  tour  of  duty,  the 
ensign  is  envisaged  as  being  assigned  to  a watch  as  Officer  of  the  Day  (00D). 
Item-ideas  relating  to  this  situation  involved  various  problems  of  ship-maneuvering, 
communications,  CIC,  anti-submarine  warfare,  etc. 

Inspection  of  the  items  resulting  from  this  mode  of  attack  shows  that  many  of 
them  must  frankly  be  regarded  as  measuring  mere  factual  information.  For  one 
thing,  having  relevant  factual  information  is  doubtless  crucial  in  many  of  the 


- 16  - 


situations  on  which  these  items  are  based.  Secondly,  in  an  examination  it  is 
hardly  possible  to  present  situations  in  all  the  detailed  context  in  which  they 
would  occur  in  real  life.  Thirdly,  instructors  at  Newport  insist  that  the 
curriculum  at  OCS  is  of  such  a nature  that  they  cannot  give  students  any  real 
understanding  or  impression  of  the  way  things  happen  in  shipboard  operations! 
a student  can  get  the  "feel"  of  being  a naval  officer  only  after  he  has  had 
experience  in  the  fleet.  Therefore,  any  questions  which  might  probe  into  the 
subtler  and  more  complicated  aspects  of  fleet  activities  would  be  almost  mean- 
ingless to  officer  candidates.  Nevertheless,  to  the  extent  that  the  items 

• v 

4 

which  appear  in  the  final  form  of  the  March  1953  Operations  examination  are 
embedded  in  somewhat  realistic  contexts,  they  may  cause  the  examinee  to  answer 
not  merely  in  terms  of  bookish  information  but  in  terms  of  his  understanding, 
however  dim,  of  the  way  in  which  this  information  is  relevant  in  actual  fleet 
operations. 

Thus,  the  Officer  Candidate  School,  Newport,  was  provided  with  new  achieve- 
ment examinations  for  each  of  its  six  areas  of  instruction.  These  examinations 
have  now  been  reproduced  in  quantity  by  OCS  Newport  and  are  designed  to  be  ad- 
ministered to  future  graduating  officer  candidate  classes  for  some  time  to  come, 
i.e.,  until  new  examinations  are  prepared.  The  total  number  of  items  in  each 


examination  is  as  follows: 

Orientation  and  Military  Justice 75  items 

Operations 50  items 

Seamanship 50  items 

Navigation .60  items 

Naval  Weapons 75  items 

Engineering  and  Damage  Control 75  items 
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! B.  Construction  of  Achievement  Examinations  for  Midshipmen  (NROTC) 

It  was  established  that  the  NROTC  1952-53  and  OCS  1952  curricula  were  not 
completely  comparable  with  respect  to  the  subject-matter  taught,  a situation  which 
made  the  construction  of  common  examinations  extremely  difficult,  if  not  impossible. 
NROTC  trainees  are  given  a more  extensive  course  involving  relatively  more  theory, 
whereas  OCS  trainees  receive  a course  which  emphasizes  practical  aspects.  Moreover, 
many  items  of  instruction  presented  to  NROTC  trainees  are  omitted  altogether  at 
OCS. 

To  illustrate:  the  OCS  curriculum  is  subdivided  into  six  areas  of  instruction, 

as  already  explained.  The  NROTC  curriculum  is  subdivided  into  four  college  years 
of  instruction,  each  of  the  years  cutting  across  the  six  OCS  areas  of  instruction. 

In  the  first-year  NROTC  program,  39  class  periods  are  devoted  to  the  study  of 
Naval  History,  but  in  the  Orientation  course  at  OCS,  Naval  History  is  not  formally 
taught,  the  trainees  being  required  to  read  outside  of  class  a text  on  Naval 
History.  Thus,  great  emphasis  is  placed  on  this  subject  in  NROTC  whereas  it  appears 
to  be  only  incidental  to  the  OCS  curriculum.  In  fact,  questions  on  naval  history 
have  been  completely  eliminated  from  the  OCS  Orientation  and  Military  Justice 
examination.  Conversely,  certain  subject-matter  in  Operations  such  as  CIC  and 
communications  is  more  heavily  emphasized  at  OCS  than  in  NROTC  training,  where 
there  is  considerable  tendency  to  avoid  teaching  information  based  on  classified 
documents. 

The  celestial  problem  in  the  OCS  Navigation  examination  was  constructed  to 
conform  with  the  1952  Nautical  Almanac  in  use  at  OCS.  However,  it  was  necessary 
to  reconstruct  the  whole  problem  for  the  NROTC  examination  because  midshipmen  .. 
were  uo  be  provided  with  the  1953  Almanac. 

The  construction  of  identical  examinations  for  OCS  and  NROTC  in  Naval  Weapons 
was  impossible.  The  two  curricula  are  not  comparable,  at  least  from  a test 
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construction  point  of  view.  In  fact,  great  difficulty  was  experienced  at  the  out- 
set by  project  personnel  in  constructing  items  for  incorporation  in  both  examina- 
tions. The  opinions  of  NROTC  subject-matter  instructors  differed  greatly  from 
those  of  OCS  consultants  on  which  items  were  valid  and  fair.  Items  suggested  by 
NROTC-  consultants  were  vigorously  opposed  by  OCS  consultants  and  vice  versa.  The 
primary  basis  for  disagreement  was  difference  in  the  two  curricula.  Certain 
materials  presented  in  the  NROTC  program  are  not  taught  at  OCS,  for  example, 
radar  in  connection  with  gunnery,  and  atomic  theory. 

Thus,  it  was  evident  that  examinations  constructed  for  OCS  use  would  be  in- 
adequate for  the  NROTC  program,  both  in  length  (total  number  of'  items  per  exam- 
ination) and  scope.  It  was  therefore  decided  to  assemble  new  examinations  for 
the  NROTC  program. 

Essentially  the  same  procedure  as  that  employed  in  assembling  the  OCS  exam- 
inations was  followed  in  constructing  examinations  for  NROTC,  Test  outlines 
based  on  the  1952-53  curriculum  were  designed  for  each  of  the  four  Naval  Science 
(NROTC)  examinations.  NROTC  instructors  at  Harvard  University  and  Tufts  College 
were  consulted  for  their  views  on  the  outlines  specifically  with  respect  to  cur- 
riculum coverage  and  areas  of  emphasis.  The  test  outlines  were  modified  in 
accordance  with  their  suggestions  pertaining  to  areas  of  emphasis  wherever 
necessary. 

On  the  basis  of  item  analysis  data,  valid  and  applicable  items  from  the 
newly  constructed  OCS  examinations  and  former  NROTC  examinations  were  selected 
and/or  revised  for  inclusion  in  the  NROTC  examinations.  Additional  items  were 
constructed  in  collaboration  with  Navy  subject-matter  consultants  wherever 
necessary  to  complete  the  coverage  indicated  by  the  test  outlines.  Items  to  be 
incorporated  in  each  of  the  four  new  NROTC  examinations  were  then  reviewed  by 
NROTC  instructor  personnel.  Further  item  revisions  were  made  as  necessary. 
Throughout  this  procedure,  as  before,  an  attempt  was  made  to  select  or  to 
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construct  items  of  the  desired  "integrated"  type,  but  it  cannot  be  said  that  this 
attempt  was  completely  successful  in  view  of  the  large  amounts  of  informational 
material  which  had  to  be  covered.  For  example,  little  could  be  done  to  alter  the 
general  nature  of  the  items  testing  Naval  History  in  the  First  lear  NROTC  examination. 

The  items  for  each  of  the  four  achievement  examinations,  together  with  -per- 
tinent item  analysis  data,  were  forwarded  to  the  Bureau  of  Naval  Personnel  for 
final  review  prior  to  the  typing  of  reproduction  copy* 

The  total  number  of  items  for  each  of  the  Officer  Candidate  Achievement 


Examinations  (NROTC)  is  as  follows.* 

Naval  Science  I (Freshmen) 150  items 

Naval  Science  II  (Sophomores) 150  items 

Naval  Science  III  (Juniors) .105  items 


Part  1-75  items 
Part  2 — 30  items 

Naval  Science  IV  (Seniors) 150  items 

Final  reproduction  copy  of  the  Officer  Candidate  Achievement  Examinations 
(NROTC)  and  an  Examiners  Manual  (modeled  on  those  provided  for  previous  years) 
for  the  administration  and  scoring  of  the  examinations  were  submitted  to  the  Bureau 
of  Naval  Personnel,  Washington,  D.C,  for  reproduction  and  distribution  to  the  52 
college  NROTC  units.  Prepunched  scoring  keys  for  the  examinations  were  also 
supplied  to  BuPers  in  sufficient  quantity  for  direct  distribution  to  the  NROTC 
units. 

Summary  and  Discussion  of  Phase  I of  the  Project 

Phase  I was  concerned  chiefly  with  the  provision  of  a new  set  of  examinations 
to  be  administered  at  NROTC  units  in  May,  1953.  Certain  desiderata  for  these 
examinations  were  specified  by  the  project  sponsor,  chiefly  to  the  effect  that 
the  examinations  should  test  "integrated"  knowledge  rather  than  mere  knowledge  of 
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isolated  facts#  To  the  extent  that  it  Was  possible  to  do  so  in  view  of  the  target 
dates  for  completion,  thit  objective  was  achieved.  A new  set  of  examinations  for 
NROTC  was  provided,  and  considerable  progress  was  made  in  developing  items  calling 
for  problem-solving,  decision '-making,  and  weighing  of  factors  in  complex  situa- 
tions. In  the  course  of  preparing  examinations  for  NROTC,  a new  set  of  examina- 
tions for  OCS  was  prepared. 

::  It  is  pertinent  to  comment  on  the  extent  to  which  the  approach  described 

above  was  found  practicable.  This  will  be  done  by  considering  each  of  several 
•’  areas  of  instructions. 

Nearly  all  the  subjects  covered  under  Naval  Operations,  including  tactics, 
maneuvering  board,  communications,  CIO,  and  ASW,  are  amenable  to  the  construction 
of  problem-solving  and  decision-making  items.  The  project  made  considerable 
progress  in  this  direction,  as  may  be  seen  by  inspection  of  the  March,  1953  OCS 
- Operations  examination  (NAVPBRS  18307,  March  1953,  available  at  OCS  Newport). 

However,  it  was  found  impossible  to  include  "integrated"  type  items  on 
i:  Operations  in  the  NROTC  examination,  since  the  subject-matter  in  this  area  taught 
-in  NROTC  is  of  a relatively  elementary  character.  The  only  items  of  the  problem- 
solving type  included  in  NROTC  examinations  were  maneuvering  board  problems  (see, 
for  example,  questions  6l  - 70  in  the  Third  Tear  NROTC  Examination,  Part  I, 

NAVPERS  18288,  May  1953). 

. Of  the  subjects  taught  under  Orientation  and  Military  Justice,  those  having 

to  do  with  military  insignia,  naval  regulations,  naval  customs,  the  naval  estab- 
lishment^ naval  administration  and  correspondence,  and  naval  history  were  in 
general  found  not  amenable  to  the  construction  of  problem-solving  or  decision- 
making items.  Most  of  the  items  of  knowledge  taught  in  this  area  exist,  at  least 
for  beginning  trainees,  as  isolated  bits  of  information  and  cannot  be  thrown  into 
any  under  lying  rationale  or  structure.  While  it  is  conceivable  that  problem- 
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solving  situations  invely ing - the b e.  f ae  t a could  be~ccnstructe«l  as  examination  items, 
the  effort  required  and  the- space  needed  on  the  examination , paper  could  hardly  be 
justified.  A more  conventional,  direct  approach  has  been  deemed  preferable  in 
testing  these  subject-matters*  Subject-matter  in  naval  law  and  naval  leadership, 
however,  lends  it self -more -readily  to  the  problem-solving  approach.  Naval  law 
includes  much  factual  information  which  must  be  presented  and  tested  a8  such,  but 
at  the  same  time  presents  possibilities -for  constructing  items  which  depict  some  of 
the  types  of  situations  which  a naval  officer  might  commonly  encounter.  Attempts 
in  this  direction  are  represented  by  items  110-113,  121,  and  125  on  the  Fourth  Tear 
NROTC  Examination  (HAMPERS  18290,  May  1953) • However,  the  situations  cannot  be  so 
complex  that  they  present' technicalities  which  can  be  solved  only  by  an  officer 
with  special  legal  training. 

-Likewise,  the  subject-matter  of  Naval  Leadership  lends  itself  to  items  of  the 
situational  type.  An  especially  good  example  is  provided  by  item  132  of  the  above- 
cited  Fourth  Year  NROTC  examination:  (NAVPERS  18290,  May  1953).  This  item  was 
especially  constructed  by  project  personnel  to  test  knowledge  of  certain  principles 
of  leadership  and  ability  to  weight  various  factors  in  the  situation;  to  our  know- 
ledge, no  situation  exactly  like  this  was  discussed  in  NROTC  classrooms.  (This 
does  not  preclude  the  possibility  of  "coaching"  in  future  years,  of  course.) 
Unfortunately,  the  validity  of  this  item  is  unknown  since  it  was  included  only  in 
the  May  1953  NROTC  examination,  given  too  late  to  be  item-analyzed  by  this  project. 

The  subject  of  Navigation  is  obviously  amenable  to  testing  by  problem-solving 
items.  In  fact,  a major  section  in  previous  examinations  had  been  constructed  in 
this  form,  and  it  was  only  necessary  for  this  project  to  adapt  this  section  to  use 
with-  several  successive  editions  of  tho  Nautical  Almanac.  The  reader  will  recall 
that  this  section  (Part  II,  items  31-60)  consists  of  30  items  based  upon  a "day's 
work"  in  which  the  examinee  is  required  to  solve  typical  navigational  problems. 
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There  remain,  of  course,  many  aspects  of  the  subject-matter  of  navigation  v'.ich 
do  not  l6nd  themselves  conveniently  to  problem-solving  items.  For  example, 
questions  regarding  the  meaning  of  map  symbols,  the  classification  of  buoys, 
navigational  publication*:,  and  the  like  can  be  best  handled  by  direct  questions 
designed  to  measure  factual  information.  Such  questions  are  found  among  the  first 
thirty  items  on  the  OCS  Navigation  examination  (NAVPERS  18309). 

A similar  situation  exists  in  connection  with  the  subject  matter  of  Seamanship. 
Items  on  ship  nomenclature,  ships  of  the  Navy,  and  many  aspects  of  deck  seamanship 
(e.g.,  "What  is  the  length  of  a standard  shot  of  anchor  chain?r)  could  conceivably 
be  set  in  "integrated,  problem-solving"  form,  but  they  are  more  easily  constructed 
as  single,  direct  multiple -choice  items.  On  the  other  hand,  topics  such  as 
weather,  the  duties  of  the  OQD,  rules  of  the  road,  and  the  like  lend  themselves 
to  problem  situations.  The  project  made  some  progress  in.  constructing  such  items, 
but  more  could  be  done  along  these  lines. 

As  has  been  mentioned  above,  a major  effort  01  the  project  was  directed 
towards  the  construction  of  "integrated"  items  in  Naval  Weapons.  The  approach 
was  to  present  typical  situations,  and  then  ask  questions  relating  to  decisions 
required  in  these  situations.  In  the  opinion  of  project  personnel,  the  examination 
in  Naval  Weapons  now  in  use  at  OGS  (NAVPERS  I83IO)  represents  a close  approximation 
to  the  best  that  can  be  done  in  the  light  of  the  present  curriculum.  For  example, 
items  10-13  therein  present  a situation  in  which  the  examinee  is  to  assume  that 
he  is  assigned  to  the  main  battery  of  a Baltimore  class  heavy  cruiser  which  is 
engaged  in  a night  surface  action  with  enemy  heavy  cruisers."  The  examinee  is 
required  to  indicate  the  correct  decisions  in  the  following  cases; 

(item  10)  which  type  of  projectile  to  use  to  inflict  maximum  belov- 
deck  damage) 

(item  11)  which  radar  spotting  method  to  use; 
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(Item  12)  how  the  r&ngekeeper  should  be  adjusted  for  differing  generated 
range  and  observed  range j 

(item  13)  how  battery  control  should  be  shifted  after  the  forward 
- controlling  director  suffers  a casualty. 

Nevertheless,  even  in  situational -type  items  it  was  found  necessary  to  resort 
to  questions  of  a more  factual  nature,  in  order  t cover  certain  theoretical  aspects 
of  the  curriculum.  For  example.,  item  8 in  NA7PERS  18310  begins  with  the  Following 
lead:  "The  depth  charge  attack  fails,  lour  ship  therefore  launches  a hedgehog 

attack  using1  a 7,2"  projector  charge.  This  charge  derives  its  propulsive  force 
from  . , ."  The  question  thus  becomes  quite  factual  and  specific.  It  would 
'prdbably  be  undesirable  to  eliminate  such  questions  altogether. 

The  subject-matter  of  Engineering  and  Damage  Control  was  not'  adequately 
•investigated  by  the  project  from  a test -construction  point  of  view,  due  to  lack 
bf  time'and  personnel.  -'The  final  forms  of  examinations  covering  this  subject 
cannot  be  said  to  contain  any  large  number  of  items  of  a problem-solving  type. 

The  impression  was  gained,  however,  that  much  could  be  done  in  the  direction  of 
constructing  tests  Of  integrated  knowledge  in  this  area.  This  is  true  despite  the 
fabtthat  the  curricula  of  both  OCS  and  NROTC  require  the  presentation  of  much 
theoretical  knowledge  concerning  such  hatters  as  the  physics  of  pressure,  heat, 
and  gravity:  characteristics  of  boilerdj  engines)  and  the  like.  It  is  envisaged 
that  in  further  test-construction  efforts  in  this  area,  items  could  be  presented 
in  the  context  of  typical  engine-room  and  damage-control  happenings.  Questions 
could  be  asked  about  measures  to  be  taken  in  various  types  of  emergencies.  Questions 
about  ship-loading  also  would  lend  themselves  to  the  use  of  the  problem-solving 
approach, 

'•'During  the  test-construction  phase  of  this  project,  the  staff  found  itself 
constantly  searching  for  tangible  ways  by  which  one  could  be  assured  that  exam- 
ination questions  of  the  desired  "integrated"  type  were  being  constructed. 
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Although  it  was  possible  to  identify  items  which  the  staff  could  agree  upon  as 
appearing  to  be  of  the  desired  type,  subjective  agreement  was  not  a scientifically 
satisfying  criterion.  Phase  X of  the  project  was  thus  concluded  with  only  a hope 
and  a promise:  somewhere  in  the  wide  variety  of  items  which  had  been  constructed, 

items  of  the  desired  type  could  be  reasonably  expected  to  exist.  It  was  believed, 
however,  that  these  items  could  be  objectively  identified  only  by  resort  to 
statistical  factor -analytic  procedures.  Such  procedures  would  presumably  separate 
several  types  of  items,  one  or  more  of  which  could  be  appraised  as  items  t ? an 
"integrated”  type,  others  of  which  could  be  considered  as  more  "factual"  in 
nature.  For  this  reason,  at  least  half  the  total  time  of  the  project  was  devoted 
to  statistical  analysis,  now  to  be  described  in  Part  II  of  this  report* 


part  two 

EXPERIMENTAL  AND' STATISTICAL  ANALYSES 
r - CHAPTER  III.  GENERAL  STATISTICAL  EVALUATION  OF  EXAMINATIONS 

Introduction  • - >. 

Before  proceeding  to  more  advanced  phases  of  statistical  analysis,  the  project 
staff  desired  to  appraise  the  extent  to  which. the  examinations  constructed  in  Phase  I 
met  the  usual  criteria  of  good  tests — appropriate  forms  of  score  distribution,  high 
reliabilities,  and  the  like, — and. also  to  appraise  in  a preliminary  way  whether  the 
; v;a^tQUs  tests,  organized  chiefly  in  terms  of  subject-matter,  measured  different 
aspects  of  -achievement.  . . ; 

All  the  analyses  reported  in  this  chapter  are  based  on  examinations  prepared 
for  and  administered  to  various  classes  of  officer  candidates  at  the  U.  S.  Naval 
School  (Officer  Candidate),  Newport,  Rhode  Island.  In  the  case  of  the  examinations 
prepared  for  administration  to  NROTC  students  in  May  1953,  data  could  not  be  obtained 
soon  enough  for  analysis  before  the  expiration  of  the  contract.  Since  the  NROTC 
examinations  are  generally  similar  in  form  to  the  OCS  exams,  it  is  probable  that 
certain  generalizations  about  the  OCS  examinations  can  be  extended  to  apply  to 
NROTC  examinations. 

Teat  Score  Distributions 

One  of  the  first  things  which  had  to  be  determined  was  whether  the  time-limits 
provided  for  the  examinations  were  adequate  to  allow  nearly  all  examinees  to  finish 
trying  all  items.  We  espouse  the  point  of  view  that  officer  candidate  achievement 
tests  should  measure  accuracy  of  knowledge  rather  than  speed  in  answering  test 
items.  Except  in  certain  situations  (e.g.,  solving  maneuvering  board  problems  under 
combat  conditions),  speed  in  solving  problems  is  probably  not  as  important  as 
accuracy  in  successful  performance  of  naval  officers,  and  certainly  not  in  the 
training  of  naval  officer  candidates. 
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Accordingly,  distributions  were  made  of  the  item-number  of  the  last  item 
attempted,  for  the  six  final  examinations  given  to  Class  VI  at  OCS  (this  class 
graduated  July  1952).  The  results  are  shown  in  Table  3.1,  based  on  10  randomly 
Selected  sections  of  the  class.  In  view  of  the  large  proportions  of  examinees 
completing  all  items,  it  was  concluded  that  the  previously  established  time-limits 
were  adequate.  Accordingly,  no  major  changes  in  the  length  or  time-limit  of  the 
OCS  tests  were  made  in  subsequent  revisions,  except  that  the  time-limit  for  the 
Seamanship  examination  was  increased  to  UO  minutes  in  view  of  its  lengthening  to 
50  items.  It  should  also  be  pointed  out  that  the  March,  1953  Operations  exam- 
ination contained  only  50  items  instead  of  the  previous  75.  No  change  in  time- 
limit was  made,  however,  because  the  items  were  chiefly  of  the  situational  problem 
type  and  were  expected  to  require  more  time  per  item. 

The  form  of  the  test-score  distributions  was  next  considered.  (The  score 
used  in  all  cases  was  the  number  right.*)  When  plotted  on  arithmetical  proba- 
_ bility  paper,  all.  score  distributions  were  seen  to  be  close  approximations  to 
normal  distributions.  It  was  not  considered  worthwhile  to  apply  exact  statistical 
tests  of  normality.  In  view  of  the  approximate  normality  of  distributions  and  the 
fact  that  the  means  and  standard  deviations  are  presented  in  Table  3.2,  it  is  also 
not  considered  worthwhile  or  necessary  to  present  here  the  graphs  showing  the 
exact  form  of  each  distribution. 

Test  Means,  Standard  Deviations,  and  Reliabilities 

The  data  to  be  discussed  in  this  section  are  presented  in  Table  3.2,  showing 
various  test  statistics  for  each  of  the  numerous  examination  forms  administered 
to  the  classes  VI  to  IX  at  OCS  Newport.  Test  forms  in  each  subject  are  indicated 
by  A,  B,  and  C,  together  with  primes } use  of  letters  B or  C indicates  major 

The  score  in  the  Navigation  Examination  was,  however,  arrived  at  by  the 
Newport -established  formula  which  weighted  Part  I by  a factor  of  2 and 
Part  II  by  a factor  of  3. 

rv  *-» 
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TABLE  3.1 

DATA  ON  NUMBER  OF  ITEMS  ATTEMPTED  IN 
SIX  OCS  EXAMINATIONS,  CLASS  VI 


Time-limit  No.  of  Distribution  of  Klast  item 
(minutes)  Items  attempted11 


Orientation  and 
Military  Justice  60 


Operations 


All  examinees  completed 
75  items. 


? — 

f 

70 

1 

69 

0 

68 

1 

67 

1 

66 

2 

Seamanship 

36 

U5  All  examinees  completed 

bS  items* 

Navigation  Part  I 


All  examinees  completed 
30  items 


Navigation  Part  II 


30  30  223 


Naval  Weapons 


Engineering  and 
Damage  Control 


All  examinees  completed 
75  items. 


TABLE  3.2 


MEANS,  STANDARD  DEVIATIONS,  AND  RELIABILITY  AND  HOMOGENEITY  DATA 

FOR  OCS  EXAMINATIONS 

Note:  All  data  on  reliability  and  homogeneity  are  based  on  random  samples 

of  200  cases  used  for  item  analysis. 


K-R 

Class  and 

Test 

Max. 

(20) 

rel. 

tj 

Date  Graduated 

N Form 

Score 

X 

a 

rel. 

(1  item) 

Ht 

Engineering  and  Damage  Control 


VI 

(July  1952) 

681 

A 

75 

5U.5 

6.9 

* 

3C 

VII 

(Oct.  1952) 

68b 

A* 

75 

50,0 

7.0 

.706 

.on 

,0U9 

VIII  (Dec.  1952) 

772 

A’ 

75 

[48,8 

7.U 

.7u3 

.039 

.059 

IX 

(March  1953) 

83)- 

A' 

75 

U8,3 

7.1 

# 

Navigation  (Part  I 

only) 

VI 

(July  1952) 

681 

A 

30 

20.2 

3.1 

* 

* 

* 

VII 

(Oct.  1952) 

68ii 

B 

30 

22.1** 

3.1** 

.513 

.03U 

.063 

VIII 

(Dec.  1952) 

772 

B> 

30 

20, 0** 

3.0** 

.U99 

,032 

.061 

IX 

(March  1953) 

83U 

B« 

30 

-V- 

•M. 

r- 

Navigation  (Part.  II  only) 

VI 

(July  1952) 

681 

A 

50 

~ n 

- - 0 ’ 

3.8 

•ft 

VII 

(Oct.  1952) 

68L 

B 

30 

2 3,5** 

3 . 

. 782 

.107 

,155 

VIII 

(Dec.  19.52) 

772 

p.t 

JC 

■ 7 7 - 

C-  C , • ‘ ■ 

3.3^ 

,696 

,071 

.I8I4 

IX 

(March  1953; 

83L 

B* 

30 

•*- 

->?■ 

Navigation  (Parts 

I ar.d 

II  c 

ombir.ecu 

wi  rn  par 

’ I we: 

.glued  by 

2 and  par.  '• 

II  we: 

ighted  by  3 

1 

0 / 

VI 

(July  195?) 

681 

A 

150 

V. 

>«. 

* 

VII 

(Oct.  1952) 

681 

B 

150 

1 ’ 1 - 7 

! t-0 

. 7?r+ 

VIII 

(Dec.  1952) 

772 

35 

150 

112.2 

■*■3 , 3 

7 I C4r 
0 „*  ■ 

*'• 

IX 

(March  1953) 

83/4 

B1 

i5o 

11/4.7 

13,9 

”• 

»c 

■>?• 

Operations 

VI 

(July  1952) 

681 

A 

75 

u?  »u 

6,0 

V- 

* 

VII 

(Oct.  1952) 

681+ 

B 

75 

L8.7 

60  u 

,685 

,0:6 

.Cu8 

VIII  (Dec,  1952) 

772 

B 

7 5 

j8„2 

6,2 

^ w*.  7 

,026 

.0/48 

IX 

(March  1953) 

83U 

r> 

O 

00 

31 , ? 

8 , ‘~ 

7 635. 

.033 

.on 

(Table 

continued  on  next 

page) 
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TABLE  3.2  (continued) 


Class  and 
Date  Graduated 

N 

Test 

Form 

Max. 

Score 

I 

a 

K-R 

(20) 

rel. 

rel. 

(1  item) 

Ht 

Orientation  and  Military  Justice 
VI  (July  1952)  681  A 75 

5i.o 

5.1 

* 

* 

•a 

VII  (Oct.  1952) 

684 

B 

75 

52.0 

5.2 

.533 

.015 

.027 

VIII  (Dec.  1952) 

772 

B 

75 

51.7 

5.2 

.546 

.016 

.029 

IX  (March  1953) 

831; 

B 

75 

51.6 

5.1 

* 

-a- 

* 

Seamanship 

VI  (July  1952) 

681 

A 

45 

34.2 

3.3 

* 

* 

a 

VII  (Oct.  1952) 

681; 

B 

50 

39.6 

3.1; 

.419 

.014 

.028 

VIII  (Dec.  1952) 

772 

B‘ 

50 

35.0 

4.3 

.541 

.023 

.041 

IX  (March  1953) 

831; 

B» 

50 

35.8 

4.0 

* 

* 

* 

Naval  Weapons 

VI  (July  1952) 

681 

A 

75 

54.5 

6.5 

a- 

* 

a 

VII  (Oct.  1952) 

681; 

B 

75 

52.0 

6.2 

.683 

.028 

.051 

VIII  (Dec.  1952) 

772 

C 

75 

U7 « 7 

7.3 

.746 

.038 

.064 

IX  (March  1953) 

831; 

C 

75 

45.2 

7.1 

* 

* 

* 

* Not  computed. 

Based  on  the  random  sample  of  20C  cases  used  for  item  analysis. 

+ Estimated  by  Gulllksen's  formula  (74)  in  Chapter  20  (Gulliksen,  H, 
Theory  of  Mental  Tests,  New  York:  Wiley,  195c) . The  correlation 

between  Part  I and  Part  II  of  the  Navigation  examination  was  .27 
in  Glass  VII } this  figure  was  used  as  an  estimate  of  the  correlation 
in  Class  VIII,  which  was  not  actually  computed. 
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revisions  from  For*  A,  while  a prime  indicates  a minor  revision.  For  example, 
the  Seamanship  examination  Class  VII  represented  a considerable  revision  (Form  B), 
but  underwent  only  minor  revisions  thereafter  (Form  B‘).  Allusions  will  also  be 
made  to  Table  3.3  which  concerns  the  statistics  of  academic  department  grades  at 
Newport. 

With  regard  to  test  means  and  standard  deviations,  the  following  interpretations 
may  be  made: 

(a)  The  test  mean  is  anywhere  from  60$  to  ?8/£  of  the  total  possible  score. 

This  is  mentioned  chiefly  because  of  the  practice  at  Newport  of  converting  the 
test  scores  to  the  Navy  U.O  system  of  grading  by  the  formula 


Navy  grade  = k 


/ Sc 
( Max. 


Score 


Score 


with  the  intention  of  obtaining  an  average  Navy  Grade  of  about  3.0  to  3.2,  If 
the  tests  are  to  be  used  with  this  procedure  in  mind,  the  test  mean  should  be 
from  75  to  80  per  cent  of  the  total  possible  score.  Nevertheless,  the  Navy 
formula  has  the  disadvantage  of  assuming  that  tests  measure  in  terms  of  absolute 
standards.  The  characteristics  of  a test  score  distribution  should  be  evaluated 
from  the  standpoint  of  relative  rather  than  absolute  standards.  If  tests  of 
maximum  reliability  are  desired,  one  guiding  principle  is  that  the  test  mean 
should  be  about  in  the  center  of  the  range  from  a pure  chance  score  to  the  maximum 
score,  i.e.,  about  60  per  cent  of  the  total  possible  score  for  tests  with  5-choice 
items.  In  view  of  this,  the  OCS  examinations  reported  on  in  Table  3.2  are 
slightly  too  easy.  It  shcrild  be  remembered  that  the  ratio  between  test  mean  and 
the  maximum  score  is  directly  related  to  the  average  item  difficulty.  In  a later 
section  of  this  report,  the  distributions  of  item  difficulties  will  be  discussed, 
(b)  There  is  some  tendency  for  the  test  means  (at  least  in  terms  of  average 
item  difficulties)  to  decrease  as  one  moves  from  Class  VI  to  Class  IX,  This  is 
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alec  reflected  in  the  mean  Bureau  Examination  averages  reported  in  Table  3,3  which 
decrease  from  2 • Sip.  in  Claee  VII  to  2,706  in  Class  IX  (the  difference  being 
significant  far  beyond  the  *001  level).  This  result  may  be  due  to  decreasing 
quality  in  the  successive  classes  or  to  the  failure  of  the  examinations  to  keep 
pace  with  gradual  changes  in  the  curriculum.  For  example,  the  mean  test  score 
in  the  Engineering  and  Damage  Control  examination,  which  wa3  subjected  to  only 
very  minor  revisions  by  this  project,  decreased  from  to  1*8.3  from  Class  VI 
to  IX.  (This  difference  is  also  significant  far  beyond  the  .001  level.)  On  the 
other  hand,  the  decrease  is  at  least  partly  due  to  the  fact  that  in  successive 
revisions  of  the  various  examinations  the  project  staff  sought  to  avoid  very 
easy  items  (particularly  when  they  showed  low  validities),  and  in  general  to  reduce 
the  average  item  difficulty  index  (proportion  passing).  It  also  appears  that  when 
attempts  were  made  to  subject  tests  to  major  revisions  (to  the  extent  of  constructing 
numerous  new  items  of  a situational  problem  type),  the  average  item  difficulty  and 
hence  the  test  mean  tended  to  be  reduced.  This  tendency  seems  to  be  illustrated 
in  the  case  of  the  Naval  Weapons  and  the  Operations  examinations.  (It  would  have 
been  possible  to  test  the  significance  of  the  decreases  only  by  use  of  analysis 
of  covariance  with  suitable  control  variables  such  as  Officer  Candidate  Batteiy 
scores,  but  this  was  not  considered  worth  the  necessary  effort.  Furthermore, 
this  procedure  would  be  rather  unusual  in  using  different  dependent  variables  for 
the  various  classes.) 

(c)  The  test'  standard  deviations  are  such  that  +3 a from  the  mean  covers 
anywhere  from  $1  to  78  per  cent  of  the  range  from  a pure  chance  score  to  a perfect 
score.  It  was  thought  desirable  to  increase  these  ranges  with  successive  test  re- 
visions by  appropriate  selection  of  items,  and  there  is  some  evidence  that  the  project 
was  successful  in  this  regard,  for  example  in  the  Operations  and  Naval  Weapons  exam- 
inations. Further  progress  in  this  direction  could  in  theory  be  made  by  selecting 
items  with  a wider  range  of  difficulty  and  with  somewhat  higher  validities. 
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(d)  Eeliabilities  of  many  of  the  examinations  were  computed  from  item- 
analysis  data  by  means  of  the  Kuder -Richardson  formula  (20).*  Since  the  July  1952 
examinations  and  most  of  the  March  1953  examinations  (the  single  exception  being 
Operations , which  waw  a completely  new  examination)  were  not  subjected  to  item 
analysis,  no  reliability  data  for  these  are  reported  in  Table  3.2, 

The  reliabilities  of  the  examinations  were  disappointingly  low,  in  spite  of 
persistent  efforts  to  select  or  to  construct  items  which  would  be  valid  when 
judged  against  internal  consistency  criteria.  It  is  of  interest  to  note  that  one 
of  the  tests  least  worked  on  by  the  project  remained  one  of  the  most  reliable, 
namely,  the  Engineering  and  Damage  Control  examination,  which  had  reliabilities 
of  .706  and  .71*3,  respectively,  in  Classes  VII  and  VIII.  Taking  the  reliabilities 
of  the  latest  examinations  constructed  by  the  project,  one  may  rank  the  tests 
as  follows: 


Reliability 

Naval  Weapons  (Form  C) 

.71*6 

Engineering  (Form  A1) 

.71*3 

Navigation,  Part  II  (Form  B’) 

.696 

Operations  (Form  C) 

.632 

Orientation  and  Military  Justice  (Form  B) 

.51*6 

Seamanship  (Form  B') 

.51*1 

Navigation,  Fart  I (Form  B') 

.1*99 

v 

This  is  presented  as  formula  (11)  in  Chapter  16  of  Harold  Gulliksen's  Theory 
of  Mental  Tests  (N.Y.,  Wiley,  1950).  This  reliability  coefficient  has  often 
been  regarded  as  a lower  bound  to  the  equivalent -form  reliability.  Lee  Cronbach, 
(Cronbach,  Lee  S.,  "Coefficient  alpha  and  the  internal  structure  of  tests." 
Psychometrika  1951,  16,  297-331*.)  however,  has  recently  shown  that  it  is  also 
the  average  stepped-up  reliability  obtainable  from  all  possible  splits  of  a 
test  into  two  parts.  In  any  case,  the  formula  seems  practicable  and  per- 
tinent to  achievement  examinations  of  the  type  studied  here. 


a 

I 

i 


4 
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This,  however,  does  not  quite  give  the  whole  story.  In  the  first  place,  the 
reliability  of  the  composite  score  on  the  Navigation  examination  (Form  B1)  may 
safely  be  estimated  at  ,715  and  thus  is  almost  as  reliable  as  the  Engineering 
and  Naval  Weapons  examinations.  In  the  second  place,'  the  reliability  figures 
given  above  are  for  differing  test  lengths.  Accordingly,  the  reliability  per 
item  (given  in  th*  next  to  last  column  of  Table  3.2)  was  computed  for  each  test 
by  applying  the  Spearman-Brown  formula  " in  reverse,"*  In  these  terms,  the  tests 


In  effect,- this  procedure  estimates  the  reliability  of  a test  of  a standard 
length,  namely  one  item.  If  r is  the  reliability  of  a test  of  n items,  the 
reliability  of  one  item  (r^)  is,  by  the  Spearman -Brown  formula. 


1 


It  can  be  shown  that  this  formula  is  very  simply  related  to  the  expression 
presented  by  Harold  Gulliksen  as  "a  function  of  test  reliability  that  is  in- 
variant with  respect  to  changes  in  test  length"  (formula  22,  Chapter  8, 
Theory  of  Mental  Tests,  N.  Y, , Wiley,  1950) . Gulliksen’ s expression 
(in  our  notation)  is: 


I 

I 

i 

and  this  is  equal  to  (l  - r, ) / r^  » The  use  of  r-^  as  an  invariant  function 
of  reliability  is  preferable  to  Gulliksen1 s expression  because  it  is  in  the 
metric  of  correlation  coefficient,  whereas  Gulliksen' s expres  ion  ranges  from 
,00  (for  perfectly  reliable  tests)  to  + • (for  tests  of  zero  reliability). 


rank  as  follows  (data  are  again  for  the  latest  test  constructed) : 


Reliability  (1  item) 

Navigation  Part  II  (Form  B1)  «071 
Engineering  (Form  A')  *039 
Naval  Weapons  (Form  C)  .038 
Operations  (Form  C)  .033 
Navigation  Part  I (Form  B' ) .032 
Seamanship  (Form  B*)  .023 
Orientation  and  Military  Justice  (Form  B)  .016 


It  appears  that  the  reliability  per  item  is  a function  of  the  extent  to  which  the 
test  measures  technical  information  involving  knowledge  of  mathematics  and 
physics.  However,  it  is  not  necessarily  related  to  the  "problem-solving"  nature 
of  the  item,  since  the  Engineering  examination,  with  a relatively  high  per-item 
reliability,  contained  only  a small  number  of  such  items.  The  tests  showing 
lowest  per-item  reliabilities  were  those  concerned  with  isolated  bits  of  non- 
technical information  related  to  the  conventions,  regulations,  and  customs  of  the 
navy  and  seamanship.  Thus,  there  seems  to  be  something  inherent  in  the  character- 
istics of  various  types  of  test  content  which  is  a factor  in  the  maximum  per-item 
reliabilities  which  can  be  obtained  in  testing  these  contents.  Since  per-item 
reliability  estimates  are  also  to  some  extent  a function  of  average  item-inter- 
correlation, the  per-item  reliability  data  can  also  be  regarded  as  reflecting 
the  content  homogeneity  within  each  of  the  several  examinations. 

In  view  of  the  relatively  low  reliabilities  of  the  separate  department 
examinations,  one  may  reasonably  be  concerned  with  the  reliability  of  the 
composite  score  computed  at  Newport  and  labeled  "Bureau  Examination  average." 

This  score  is  computed  (at  least,  before  converting  to  the  Navy  grading  system) 
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by  adding  the  raw  scores  on  the  examinations,  with  a weight  of  1 for  all  examinations 
except  Navigation  (Part  I)  and  Naval  Weapons,  which  receive  a weight  of  2,  and 
Navigation  (Part  II),  which  receives  a weight  of  3*  The  reliability  of  the  composite 
score  can  then  be  estimated  by  an  application  of  the  correlaticn-of-sums  formula. 

As  shown  by  pulliksen  (formula  7U  in  Chapter  20,  Theory  of  Mental  Tests),  this  may 
be  written 


rel. 


(wtd.  composite) 


WGW»  , 
WRW' 


where  W is  a row  vector  of  weights,  C is  the  matrix  of  intercorrelations  with  test 
reliabilities  in  the  diagonal,  and  R is  the  matrix  of  intercorrelations  with  unities 
in  the  diagonal.  This  formula  was  applied  to  the  data  for  the  Class  VII  and  Class 
VIII  examinations . using  appropriate  intercorrelations  from  Table  3.5.  Since  the 
formula  assumes  that  the  variables  to  be  weighted  are  in  standardized  form,  the 
row  vector  of  weights  was  taken  as  the  standard  deviations  of  the  weighted  scores 
which  were  summed  to  make  the  composite.  Data  for  the  two  parts  of  the  Navigation 
examination  were  combined*  thus,  the  weight  for  the  Navigation  examination  was  the 
actual  standard  deviation  of  its  own  composite  score,  (A  footnote  to  Table  3.2 
explains  the  one  approximation  that  was  involved  in  this  procedure.)  By  this 
means,  the  reliabilities  of  the  Bureau  examination  average  were  estimated  to  be 
as  follows  (standard  errors  of  measurement  in  terms  of  the  Navy  grading  system 
are  also  given) : 

Reliability  G 

* meas. 


Class  VII  (October  1952) 

.888 

.082 

lass  VIII  (December  1952) 

-891a 

.085 

In  view  of  the  total  length  of  the  battery,  these  reliabilities  are  not  as  high 
as  might  be  desired,  and  need  to  be  improved  in  further  test  construction  efforts* 
likewise,  the  standard  errors  of  measurement  are  uncomfortably  large  for  accurate 


- 36  - 


ranking  of  students*  It  is  probable  that  the  composite  could  be  made  slightly 
more  reliable  by  a different  weighting  of  the  tests j the  optimal  weights  could 
be  determined  by  the  procedure  outlined  by  Gulliksen  in  Section  12,  Chapter  20, 
Theory  of  Mental  Tests.  This  procedure  is  computationally  complex,  however,  and 
would  not  produce  a large  enough  increase  in  reliability  to  Justify  its  use  except 
as  a "inal  refinement.  A more  fundamental  solution  to  the  problem  of  greater 
reliability  of  the  composite  is  to  increase  the  reliability  of  the  subtests. 

It  should  be  remarked,  however,  that  the  present  research  has  been  concerned 
more  with  the  validity  of  the  examinations  than  with  their  reliability.  Thus, 
while  the  reliabilities  of  the  successively  modified  tests  do  not  show  an  upward 
trend,  their  subject-matter  validity  may  have  increased.  Indeed,  it  is  possible 
that  "integrated"  examinations  will  be  found  to  have  inherently  low  reliabilities. 
As  has  been  pointed  out,  however,  no  external  criterion  against  which  to  measure 
validity  has  yet  offered  itself. 

(e)  Another  approach  to  content  homogeneity  is  represented  by  Loevinger's 
coefficient  of  homogeneity,  H^.*  This  coefficient  measures  (on  a scale  from 
.00  to  1.00)  the  extent  to  which  the  variance  of  a test  approaches  the  variance 
of  what  Loevinger  calls  a "perfectly  homogenous  test"  in  which  all  tetrachoric 
correlations  between  items  would  be  unity.  Such  a test  would  also  be  perfectly 
scalable  in  Louis  Guttman's  sense.  The  writer's  experience  has  been  that  as 
applied  to  most  tests  Loevinger's  coefficient  has  very  small  values,  and  this  is 
confirmed  in  the  present  instance.  Ranking  the  latest  forms  of  the  examinations 
with  respect  to  we  get  the  following  list: 


Loevinger,  Jane.  A systematic  approach  to  the  construction  and  evaluation 
of  tests  of  ability.  Psychol , Monog. , 19k7f  61,  Whole  No,  285. 
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Navigation  Part  II 

-=t 

co 

( — i 

• 

Naval  Weapons 

:Q6h 

Navigation  Part  I 

.061 

Engineering  and  Damage  Control 

.059 

Operations 

.014-8 

Seamanship 

.Gill 

Orientation  and  Military  Justice 

.029 

The  above  ranking  is  highly  related  (p  = .82)  to  the  ranking  with  respect 
to  per -item  reliability;  hence  the  two  measures  appear  to  be  measuring  much  the 
'.  same  thing,  or  at  least  to  be  affected  by  the  same  test  characteristics.  The 
relatively  high  value  obtained  for  Navigation,  Part  II,  is  consistent  with  the 
observation  that  this  whole  test  is  concerned  with  a single  navigation  problem. 

It  is  of  interest  to  note  that  in  nearly  every  case,  later  forms  of  the 
testa  tended  to  have  higher  homogeneity  coefficients  than  earlier  forms,  even  when 
the  reliability  coefficients  did  not  increase.  If  these  increases  are  significant 
(no  statistical  tests  of  significance  are  available,  however),  they  may  be  due  to 
the  fact  that  the  later  revisions  contained  items  of  higher  average  validity. 

Means  and  Standard  Deviations  of  Newport  Academic  Grades 

The  means  and  standard  deviations  of  academic  grades,  by  department,  for 
Classes  VII,  VIII,  and  IX  are  presented  in  Table  3.-3=  Data  on  the  Bureau  Examination 
Averages  for  Classes  VI  - IX  are  also  presented.  All  these  grades  are  in  terms 
of  the  Navy  grading  system,  which  uses  a scale  running  from  0.0  to  ii.O.  The 
Bureau  Examination  Average  has  a nominal  weight  of  one -tenth  in  the  Final  Average 
Academic  Grade,  It  was  of  interest  to  determine  the  actual  weights  of  the  various 
components  in  the  Final  Average  Academic  Grade.  This  was  done  by  multiple-regression 


TABLE  3.3 


MEANS  AND  STANDARD  DEVIATIONS  OF  ACADEMIC  GRADES  BY  DEPARTMENT 
AND  THE  AVERAGE  OF  BUREAU  EXAMINATIONS 

jte:  All  grades  represented  here  are  in  terms  of  the  Navy  system. 

(Scale  from  0.0  to  U.O) 


Class  and  Graduation  Date 

N 

X 

<T 

Final  Average  Academic  Grade 

VII  (Oct.  1952) 

68U 

3.089 

.182 

VIII  (Dec.  1952) 

772 

3.0U6 

.210 

IX  (March  1953) 

63U 

3.0U1 

.197 

Engineering  Department  Grades 

VII  (Oct.  1952) 

68U 

3.06L 

.217 

VIII  (Dec.  1952) 

772 

3.051 

.276 

IX  (March  1953) 

83U 

3.063 

.232 

Navigation  Department  Grades 

VII  (Oct.  1952) 

68U 

3.196 

.281 

VIII  (Dec.  1952) 

772 

3.078 

»3U7 

IX  (March  1953) 

83U 

3.102 

.29U 

Operations  Department  Grades 

VII  (Oct.  1952) 

68U 

3.153 

.222 

VIII  (Dec.  1952) 

772 

3.175 

.21*5 

IX  (March  1953) 

83U 

3.078 

.236 

Orientation  and  Military  Justice  Grades 

VII  (Oct.  1952) 

68U 

3.111 

.159 

VIII  (Dec.  1952) 

772 

3.151 

.181 

IX  (March  1953) 

83U 

3.139 

.183 

Seamanship  Department  Grades 

VII  (Oct,  1952) 

68U 

3.090 

,186 

VIII  (Dec,  1952) 

772 

3.063 

,288 

IX  (March  1953) 

83U 

3.061 

,208 

Naval  Weapons  Department  Grades 

VII  ' (Oct.  1952) 

68U 

3.030 

.238 

VIII  (Dec,  1952) 

772 

3.022 

.258 

IX  (March  1953) 

83U 

3.03U 

.21*7 

Bureau  Examination  Average  (on  U.O  system) 

VI  (July  1952) 

681 

2.8UO 

* 

VII  (Oct,  1952) 

68U 

2.81*1 

.21*6 

VIII  (Dec,  1952) 

(72 

2,718 

, 262 

IX  (March  1953) 

83U 

2,706 

.273 

* hot  computed  ' - jy  - 


techniques  for  Class  VIII,  which  was  believed  to  be  a representative  sample  for  this 
purpose.  Results  are  shown  in  Table  3*U>  which  shows  that  the  relative  weight 
of  the  Bureau  examination  average  was  11.956*  It  is  conceivable  that  the  Bureau 
examinations  deserve  somewhat  more  weight  than  this.  Possibly  there  is  hesitation 
to  put  more  weight  on  the  Bureau  examinations  because  so  doing  would  further  de- 
press the  mean  Final  Average  Academic  Grade.  Nevertheless,  if  some  sort  of 
standard  score  conversion  formula  were  used  for  the  Bureau  examinations,  instead 
of  the  Navy  grading  system,  this  could  be  avoided.  Valid  achievement  examinations 
do  not  necessarily  have  a mean  equivalent  to  3*0  or  thereabouts  on  the  Navy  grading 
scale. 

Because  the  intercorrelations  shown  in  Table  3*5  among  the  department  grades, 
which  like  the  Bureau  Examination  average,  are  in  terms  of  the  Navy  grading  system 
do  not  have  a value  of  unity,  it  is  not  unexpected  that  the  standard  deviations 
of  Final  Average  Academic  Grades  are  somewhat  lower  than  most  of  the  standard 
deviations  of  the  department  grades.  If  it  is  desired  that  the  standard  deviation 
of  Final  Average  Academic  Grades  be  higher,  it  would  be  possible  to  achieve  this 
by  use  of  a different  formula  in  arriving  at  the  final  averages. 


Intercorrelations  Among  Bureau  Achievement  Examinations 

Intercorrelations  among  Bureau  examinations  are  presented  in  one  part  of  Table  3*5» 
for  Classes  VI  through  IX.  In  Class  VI,  the  two  parts  of  the  Navigation  examination 
were  entered  as  separate  variables,  while  in  the  remaining  classes  they  were  combined 
into  one  variable  according  to  the  weighting  formula  in  use  at  Newport. 

In  general , the  correlations  among  the  examinations  are  only  of  moderate  size. 

The  correlation  between  the  Engineering  and  Naval  Weapons  examination  is  consistently 
tfhe  highest.  It  should  be  pointed  out  that  the  various  revisions  of  the  examinations 
did  not  seem  to  affect  their  intercorrelations  to  any  significant  extent,  nor  in  any 
systematic  way. 
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TABLE  3.4 

RELATIVE  ACTUAL  WEIGHTS  OF  COMPONENTS  OF  THE 
FINAL  ACADEMIC  AVERAGE 
OCS  CLASS  VIII 


Component 

. r with 
composite 

0 

Proportional 

weight 

Engineering  Dept.  Grade 

. .852 

.061 

.052 

Navigation  Dept.  Grade 

.882 

.332 

.286 

Operations  Dent.-  Grade 

.8i|0 

.211 

.182 

Orientation  Dept.  Grade 

.759 

.037 

.032 

Seamanship  Dept.  Grade 

.781 

.123 

.106 

Weapons  Dept.  Grade 

.923 

,260 

.22L 

BuExam  Average 

.777  ■ 

.138 

,119 

.997 

(multiple  R) 
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4c  3xaxa 

... 

4c  il^xisXA 

9 9 9 

4c  tHSxft 

9 9 9 

OH  CM 

4c  co  co-J 

... 

O H Ov 
4c  co  co  co 

. . . 

CM  H VO 
4c  3-J-J 

... 

. XA  CM  Ov 
4c  CO  CO-J 

9 9 9 

-JXAcO 
4c  3 co -J 

... 

JC-XA 
4c  vO  -J  so 

9 9 9 

CO  C — H 

4=  -J-Jso 

• 99 

4c  88xH 

9 9 9 

C--JCO 
4C  NOVO  NO 

... 

VI 

.VII 

VIII 

IX  . 

VI 

VII 

VIII 

IX 

VI 

VII 

vin 

IX 
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Not  computed.  **  For  Class  VI  the  upper  correlation  is  for  part  lj  the  lower  figure 

is  for  part  2.  The  correlation  between  part  1 and  part  2 was  .27. 


Since  the  correlations  are  somewhat  obscured  by  the  rather  low  reliabilities 
of  the  variables,  the  data  for  Class  VIII  were  corrected  for  attenuation.  That  is, 
use  was  made  of  the  standard  formula  for  estimating  the  correlation  between  two 
perfectly  reliable  tests; 

r...  _1«_ 

The  results  are  shown  in  Table  3.6.  It  can  now  be  seen  that  the  various  subject- 
matters  covered  by  the  tests  are.  fairly  homogeneous.  Certainly,  though  no  factor 
analysis  has  been  made  of  these  data,  there  is  a general  factor  of  achievement 
running  through  the  tests.  Engineering  and  Naval  Weapons  would  have  the  highest 
leading  on  this  general  factor,  and  Navigation  and  Orientation  would  have  considerably 
smaller  loadings.  There  is  some  evidence  for  a group  factor  which  would  include 
Seamanship  and  Operations.  In  ary  case,  Navigation  is  the  most  independent  of  the 
examinations.  Its  relatively  high  correlation  with  Operations  (particularly  evident 
in  the,  fact  * that  Part  II  of  Navigation  correlated  most  highly  with  Operations  in 
Class  VI)  is  probably  due  to  the  fact  that  both  examinations  have  the  common  feature 
of  plotting — navigational  plotting  and  maneuvering  board  plotting. 

TABLE  3.6 

INTBKCOKKELATIONS  OF  BUREAU  EXAMINATIONS,  CORRECTED  FOR  ATTENUATION, 

OCS  CLASS  VII (GRADUATING  DECEMBER  1952) 

The  reliabilities  used  in  these  computations 


t 

are 

given  in 

the  last  column  of  this 
N = 772 

table . 

Eng. 

Nav. 

Oper. 

Orient . 

Seam. 

Weap. 

rel. 

Engineering 

- 

• k6k 

• 77k 

• .559 

.698 

.871* 

.71*3 

Navigation 

.1*61* 

- 

.610 

.509 

.1*81 

.518 

.715 

Operations 

• 77k 

.610 

- 

.766 

.816 

.791* 

.669 

Orientation 

.559 

.509 

.766 

- 

.720 

.61*5 

.51*6 

Seamanship 

.698 

.UQl 

.816 

.720 

- 

.796 

.51*1 

Naval  Weapons 

.87U 

.518 

• 79k 

.61*5 

.796 

- 

.71*6 

" 1*1*  - 


The  correlations  between  the  separate  examinations  and  the  "Bureau  examina- 
tion average^  are  also  presented  in  Table  3.5.1  The  magnitude  of  these  correlations 
(which  are  part-whole  correlations)  is  a function  of  the  weights  used  in  arriving 
at  the  composite  average  score,  as  well  as  the  inter correlations  of  the  variables, 
and  (indirectly)  their  reliabilities.  The  Naval  W'apons  examination  consistently 
shows  the  highest  correlation  with  the ‘Bureau  Examination  average  but  it  also  has 
the  highest  nominal  weight  in  the  composite. 


Intercorrelations  Between  Academic  Department  Grades 

Table  3.5  also  presents  interccrrelations  among  academic  department  grades 
(including  the  final  academic  average)  for  Classes  VII,  VIII,  and  IX.  These 
correlations  tend  to  be  higher,  on  the  average,  than  the  correlations  among  the 
Bureau  examinations.  The  reliabilities  of  the  department  grades  are  not  known, 
but  may  be  assumed  to  be  quite  high  in  as  much  as  they  are  based  on  a large  number 
of  quizzes,  examinations,  and  performance  ratings.  Ar*in,  there  is  evidence  of 
a general  factor  of  achievement,  and  again,  the  correlation  between  Engineering 
and  Naval  Weapons  is  consistently  the  highest,  no  doubt  because  both  these 
subjects  involve  engineering  and  mechanical  training.  In  almost  all  respects, 
the  pattern  of  correlations  parallels  in  a general  way  that  of  the  correlations 
among  Bureau  examinations.  It  may  be  concluded  that  achievement  at  OCS,  as 
measured  by  academic  grades,  is  a fairly  homogeneous  entity. 

Intercorrelations  Between  Academic  Grade s and  Bureau  Examinations 

Table  3.5  also  presents  intercorre3ations  between  academic  department  grades 
and  the  several  Bureau  examinations  , for  OCS  Classes  VII,  VIII,  and  IX. 

If  there  is  anything  specific  in  the  content  of  instruction  in  the  several 
departments,  it  might  be  expected  that  the  correlations  between  the  academic 
grades  for  a given  department  and  the  corresponding  Bureau  examination  would  be 


i 
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higher  than  correlations  between  non -corresponding  departments  and  examinations* 

Such  an  expectation  is  borne  out  by  the  present  data  to  only  a limited  extent. 

For  example*  the  Engineering  department  grade  ia  almQst  as  highly  correlated  with 
the  Naval  Weapons  examination  as  with  the  Engineering  examination.  The  Operations 
department  grade  is  correlated  with  all  examinations,  almost  equally^  this  is  also 
true  of  the  Seamanship  department  grade.  Only  in:  the  case  of  the  Navigation  and 
Orientation  and  Military  Justice  department  grades  do  we  find  fairly  clear  evidence 
of  specificity.  For  example,  in  Class  IX  the  correlation  between  the  Navigation 
department  grade  and  the  Bureau  examination  is  .61*,  which  surpasses  the  next  highest 
correlation  .53,  with  the  Operations  examination.  Likewise,  in  the  same  class  the 
Orientation  and  Military  Justice  grade  correlates  .61*  with  the  corresponding 
examination,  clearly  surpassing  the  correlation  of  .56  with  the  Operations  exami- 
nations. 

These  data,  then,  support  the  impression  gained  earlier  that  if  the  curriculum 
is  considered  by  departments,  the  material  can  be  classified  in  three  types: 

I  Engineering  and  Naval  Weapons  material  — involving  mechanical  and 
engineering  training. 

II  Navigation  — involving  ability  in  mathematics  (particularly  Trigonometry) 
and  graphical  plotting. 

Ill  Orientation  and  Military  Justice  — involving  ability  to  acquire 
factual  information  about  navy  life,  customs,  and  regulations. 

In  the  meantime,  subject-matter  considered  under  Seamanship  and  Operations  has  a 
somewhat  ambiguous  status.  From  the  standpoint  of  factor  analysis,  it  could 
be  said  that  these  subjects  have  a complex  factor  composition. 

The  process  of  revising  the  tests  seemed  to  have  no  systematic  effect  on  their 
correlations  with  department  grades.  If  only  the  data  for  Classes  VII  and  VIII 
had  been  considered,  it  might  have  been  said  that  the  correlations  in  Class  VIII 
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were  in  some  cases  quite  different  from  those  of  Class  VII.  and  some  credence 
would  have  been  lent  to  the  belief  that  the  revisions  of  the  examinations  were 
responsible  for  these  changes.  However,  the  data  for  Class  IX  belie  such  an 
interpretation,  for  in  most  respects  the  pattern  of  correlations  was  quite  similar 
to  that  for  Class  VII.  There  was,  however,  one  surprising  change  to  be  noted  in 
the  correlation  of  the  Engineering  department  grades  and  the  Operations  examination. 
In  Classes  VII  and  VIII,  where  Form  E of  the  Operations  examination  was  used, 
these  correlations  were  .55  and  .lil,  respectively.  In  Glass  IX,  where  a radically 
different  form  of  the  examination  was  used  (Form  C)  the  correlation  dropped  to 
.18.  No  simple  explanation  for  this  effect  seems  apparent,  however,  computations 
have  been  checked  carefully.  Furthermore,  Form  C maintained  its  previous  corre~ 
lations  with  other  department  grades. 

Item  Analyses  of  PCS  Examinations 

In  the  process  of  constructing  revised  forms  of  OCS  examinations,  item 
analyses  of  many  of  the  previously  prepared  examinations  (both  for  NEOTC  and  OCS) 
were  made  available  by  the  Bureau  of  Naval  Personnel.  Much  attention  was  paid  to 
these  item  analyris  data  wherever  there  was  a question  of  selecting  an  item  from 
a previous  examination  for  use  in  one  of  the  new  examinations.  That  is,  an 
attempt  was  made  to  avoid  very  eady  items  and  items  of  low  validity  (validity  in 
this  case  being  defined  as  the  correlation  of  the  item  with  total  score). 

Since  it  was  eventually  going  to  be  necessary  to  select  items  from  the  OCS 
examinations  for  use  in  the  May  1953  examinations,  item  analyses  were  made  of 
these  examinations  as  they  were  developed.  All  examinations  administered  to 
Classes  VII  and  VIII  were  subjected  to  this  analysis,  as  well  as  Form  C of  the 
Operations  examination  administered  to  Class  IX,  since  this  was  a new  form. 

Three  types  of  data  were  secured  on  each  item: 
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(1)  The  number  and  the  proportion  passing  the  item.  The  proportion  passing 
was  identified  as  the  Difficulty  Index, 

(2)  The  estimated  correlation  between  the  item  and  the  Bureau  examination 
average  (a  composite  score  described  previously  on  page  38)*  This 
correlation  was  determined  on  the  basis  of  Flanagan’s  procedure  in- 
volving groups  of  9,  20,  1*2,  20,  and  9 per  cent,*  Facilitating  tables 
for  these  determinations  have  been  prepared  by  Flanagan.  As  noted  in 
Chapter  II,  the  composite  score  on  the  examinations  was  selected  as  the 
best  available  criterion  of  "integrated  knowledge." 

(3)  Frequencies  of  responses  to  the  five  alternatives  in  each  item. 

The  data  resulting  from  these  item  analyses  have  been  entered  on  the  item 
cards  supplied  separately  to  the  Bureau  of  Naval  Personnel,  and  since  they  are 
scarcely  meaningful  except  when  viewed  in  conjunction  with  the  tests  themselves, 
they  are  not  included  in  detail  in  this  report.  The  results  are,  however,  summarized 
in  Table  3.7,  where  medians  and  the  10  and  the  90  percentile  points  of  the  dis- 
tributions of  difficulty  indices  (per  cent  correct)  and  item  validities  are  shown. 

The  item  analyses  were  based  on  a random  33^  sample  of  the  classes.  Thus,  the 
numbers  of  cases  included  in  each  of  the  groups  were  as  follows: 


Class  VII 

VIII 

IX 

Top  9% 

21 

2k 

25 

Next  20$ 

U6 

52 

56 

(Middle  k 2$) 

(9k) 

(105) 

(1!6) 

Lower  20$ 

k6 

52 

56 

p">ttora  9% 

21 

2k 

25 

Sample  Total 

228 

257 

278 

Class  Total 

681i 

772 

83U 

The  parentheses  in  the  above  table  indicate  that  the  middle  group  is 
not  actually  used  in  Flanagan’s  procedure. 
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'TABLE  3.7 

SUMMARY  OF  ITEM  ANALYSIS  RESULTS,  OCS  EXAMINATIONS 

...  (classes  .ra,  mi,  and  u) 


Examination 

Class 

Form 

Difficulty  Index 

Ite.n 

Validity 

No.  of 
Items 

^0 

ndn 

^0 

*10 

Kdn 

P90 

Engineering 

m " 

A* 

.01 

.59 

.87 

.08 

.30 

.51 

75 

Engineering 

mi 

A* 

.01 

.66 

.91 

.06 

.25 

.38 

75 

Navigation  Part  I 

VII 

B 

.02 

.81 

.95 

.10 

.22 

.38 

30 

Navigation  .Part  1 

VIII 

B» 

.38 

.80 

.96 

.06 

.26 

.38 

30 

Navigation  Part  II 

VII  :• 

B 

.37 

.87 

.99 

• .16 

.36 

.50 

30 

Navigation  Part  II 

VIII 

B* 

.38 

.88 

.99 

.00 

.20 

.01 

30 

Operations 

VII 

B 

.314 

.69 

.93 

.03 

.26 

.38 

75 

Operations 

VIII 

B 

.32 

.71 

.93 

.02 

.20 

.33 

75 

Operations 

IX 

C 

.Uli 

.62 

.86 

.06 

.19 

.35 

50 

Orientation 

VII 

B 

.39 

.70 

.90 

-.03 

.10 

.30 

75 

Orientation 

VIII 

B 

.36 

.75 

.95 

-.00 

.10 

.26 

75 

Seamanship 

VII 

B 

.514  - 

.86 

.97 

-.03 

.15 

.37 

50 

Seamanship 

VIII 

B1 

.01 

.68 

.90 

.03 

.19 

.33 

50 

Naval  Weapons 

VII 

B 

.39 

.76 

.93 

.06 

.27 

.00 

75 

Naval  Weapons 

VIII 

C 

.30 

.67 

.90 

.09 

.27 

.05 

75 

- U9  - 


Several  points  wist  be  held  in  mind  in  considering  the  item  validity  data. 
First,  the  median  item  validities  may  be  partly  a function  of  the  weight  of  the 
examination  in  the  composite  score  which  was  used  as  the  criterion.  For  example, 
the  rather  .high  validities  found  in.  the.  Naval.  Weapons  Examination  may  result  from 
the  relatively  high  weight  ( a nominal  weight  of  2)~  of  that  examination  in  the 
composite*  Secondly,  it  cannot  necessarily  be  expected  that  the  median  item 

validity  of  a revised  test  will  be  appreciably,  higher  then  the  median  item  validity 

- ' % 

of  a-  previous  test,  even  when  item  analysis  data  were  used  in  the  revision,  in  view 
of  the  fact  that  there  is-'an  inevitable  regression  effect  due  to  sampling  errors 
in  the  item  validity  statistics.  Furthermore,  most  of  the  revisions  which  took 
place  in  the  OCS  examinations  represented  attempts  to  try  out  numerous  new  items 
never  before  used. 

It  should  be  noted  that  since  the  tests  were  either  not  at  all  speeded  or 
only  slightly  speeded,  any  tendency  for  items  near  the  end  of  the  test  to  have 
spuriously  raised  validities  due  to  time  limits  was  negligible. 

With  these  points  to  be  borne  in  mind,  Table  3»7  suggests  the  following 
conclusions:. 

(a)  The  range  of  items  in  difficulty  is  in  general  satisfactory.  For  this 
type  of  examination  it  is  felt  desirable  to  include  items  of  a wide  range  of  diffi- 
culty, in  a distribution  which  is  more  nearly  like  a rectangular  distribution 
than  a normal  distribution.  The  distributions  of  item  difficulties  in  a number 
of  the  present  examinations  are  fairly  rectangular  (in  Engineering,  Operations, 
Orientation,  and  Naval  Weapons).  Especially  in  the  Navigation  examination,  however, 
the  distribution  of  item  difficulties  tends  to  be  negatively  skewed,  with  a high 
median.  Apparently  this  subject  is  so  well  and  thoroughly  taught  that  even  the 
relatively  difficult  problems  set  in  the  examination  are  solved  by  a large  number 
of  students.  It  should  be  noted  that  despite  these  characteristics  of  the  item- 
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difficulty  distribution,  the  raw  scare  distributions  are  approximately  normal, 
and  the  tests  are  among  the  more  reliable  in  the  battery*  Indeed,  it  will  be 
recalled  that  Navigation,  Part  II  had  consistently  the  highest  per -i tom  reliability. 
These  results  only  serve  to  emphasize  again  (as  has  been  shown  in  recent  literature 
on  the  theory  of  mental  tests)  that  the  form  of  distribution  of  item-difficulties 
is  only  a small  factor  in  the  total  reliability  of  a test. 

(b)  The  median  item  validities  with  the  chosen  criterion  are  about  as  satis- 
factory as  may  t > expected  for  tests  of  this  type.  There  is  a tendency  for 
examinations  emphasizing  engineering  and  mathematical  problems  (such  as  Engi- 
neering, Navigation,  and  Naval  Weapons)  to  have  higher  median  item  validities 
than  examinations  emphasizing  factual  information  (such  as  Orientation  and 
Seamanship).  This  probably  reflects  the  fact  that  the  criterion  score  is  most 
heavily  weighted  with  variance  from  examinations  of  the  former  type.  The  fact 
that  an  item  has  a low  validity  against  the  criterion  should  not  necessarily 
discredit  the  item,  for  it  may  be  measuring  another  type  of  achievement  than 
whatever  is  represented  by  the  composite  criterion.  It  was  with  this  hope  that 
the  further  statistical  analyses  described  in  Chapter  IV  were  undertaken. 

As  a final  point  worthy  of  mention,  it  may  be  reported  that  the  project 
staff  saw  no  indication  that  item  validities  were  a function  of  item  type. 

While  some  of  the  low  item  validities  were  clearly  due  to  ambiguities  in  state- 
ment, high  validities  were  found  equally  often  for  items  appearing  to  measure 
sneer  factual  information  and  for  items  appearing  to  require  problem-solving 
and  decision  making. 
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CHAPTER  IV 


FACTOR  ANALYSIS  OF  THE  OCS  ACHIEVEMENT  EXAMINATIONS 


Introduction 

One  of  the  fundamental  questions  for  which  some  kind  of  answer  was  desired 
in  the  present  research  was  whether  the  knowledge  shown  by  officer  candidate 
examinees  at  the  end  of  their  training  is  of  a homogeneous  or  a heterogeneous 
character.  If  the  former,  it  would  be  pointless  to  hope  that  "integrated" 
examinations  (or  any  other  type  of  examination,  for  that  matter)  measuring  some- 
thing quite  different  from  the  previous  examinations  could  be  constructed,  unless 
one  makes  the  asumption  that  previous  examinations  failed  utterly  to  measure 
"integrated”  knowledge.  If  the  latter,  there  arises  the  problem  of  identifying 
the  separate  aspects  of  achievement  tested  by  the  examinations.  It  might  also  be 
hoped  that  one  or  more  of  these  separate  aspects  of  achievement  might  be  of  such  a 
nature  as  to  be  regarded  as  measuring  "integrated  knowledge"  rather  than  merely 
factual  information. 

This  chapter  reports  an  attempt  to  find  and  identify  the  separate  aspects  of 
achievement  measured  by  the  OCS  achievement  examinations.  All  data  reported  on 
here  are  based  on  the  examinations  given  to  OCS  Class  VIII  in  December  1952, 

These  examinations,  with  the  exception  of  the  Operations  examination,  (which 
was  subsequently  revised  for  the  March  1953  class),  represented  the  latest,  efforts 
of  the  project  as  far  as  OCS  examinations  were  concerned- 

selection  of  a Representative  Set  of  72  Items  for  Factor  Analysis 

The  method  selected  as  most  practicable  and  useful  for  identifying  separate 
aspects  of  achievement  was  that  known  as  the  Wherry-Gaylor  iterative  factor 
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analysis  procedure,-*  This  is  a variety  of  factor  analysis  procedure  which  is 
especially  designed  for  use  with  tests  composed  of  large  numbers  of  items.  In 
theory,  it  is  a workable  method,  but  in  practice  it  was  found  to  present  certain 
difficulties,  chiefly  difficulties  which  stem  from  the  fact  that  the  criteria  for 
determining  iterations  are  not  completely  objective  and  are  affected  unduly  by 
sampling  errors  in  item-test  correlations. 

In  the  OCS  Class  VIII  examinations,  1*10  items  in  all  were  available  for 
analysis.  In  view  of  the  fact  that  the  Wherry-Gayior  procedure  is  quite  laborious, 
it  was  decided  to  analyze  only  a representative  set  of  items.  Seventy-two  items 
were  therefore  drawn  in  such  a way  as  to  fit  into  all  of  the  72  possible  combina- 
tions of  the  following  four  dimensions: 

1,  Subject-Matter,  (Four  Categories) 

A.  Operations 

B.  Navigation 

C.  Naval  Weapons 

D„  Orientation  and  Military  Justice 

(Three  items  from  Seamanship  had  to  be  used  because  the  Navigation 
examination  did  not  yield  all  of  the  1.8  items  required  by  this 
design, ) 

2,  Item -Type,  (Two  categories) 

A.  Items  judged  by  project  personnel  to  depend  chiefly  on  memory 
of  factual  information,  terminology,  special  naval  rules  of 
procedure,  etc. 


Wherry,  R,  J,,  and  Gaylord,  R,  H,  The  concept  of  test  and  item  reliability  in 
relation  to  factor  pattern,  Psychometrika,  19l*3,  8,  21*7-269.  Unfortunately, 
a revision  of  this  procedure  was  published  too  late  for  use  in  this  project 
(Wherry,  R,  J.,  and  Winer,  B,  J,  A method  for  factoring  large  numbers  of  items, 
Psychometrika,  1953>  18,  161-179), 
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B.  Items  judged  by  project  personnel  to  depend  chiefly  on  higher  mental 
processes  such  as  reasoning,  problem  solving,  decision-making, 
comparison,  evaluation,  computation,  etc.,  with  the  memory  element 
at  a minimum. 

3.  Difficul oy.  (Three  categories) 

After  the  items  in  the  given  subject-matter  and  of  a given  item-type 
were  identified,  items  were  selected  in  three  ranges  of  item  difficulty: 

A.  Low  Difficulty 

B.  Medium  Difficulty 

C.  High  Difficulty 

U.  Item  Validity.  (Three  categories) 

The  item-analysis  data  described  in  Chapter  III  were  used  in  selecting 
items  in  three  ranges  of  item  validity: 

A.  Low  Validity  (including  near-zero  validity) 

B.  Medium  Validity 

C.  High  Validity 

The  items  were  drawn  chiefly  from  the  four  examinations  which  had  been  most 
worked  on  by  the  project  staff  and  hence  were  believed  most  likely  to  contain  a 
variety  of  items  differentiable  in  terms  of  various  aspects  of  achievement.  The 
ranges  of  item  difficulty  and  validity  values  were  set  up  after  items  had  been 
selected  for  subject-matter  and  item-type;  these  ranges  differed  slightly  among  the 
four  subject -matter  categories.  The  items  were  classified  by  item-type  at  a 
conference  of  the  three  professionals  on  the  project  staff  who  had  worked  in  the 
construction  of  the  examinations  and  who  were  hence  fairly  familiar  with  the 
content  and  nature  of  each  item. 

Table  J4..I  lists  the  72  items  selected  for  the  iterative  factor  analysis. 

Items  are  identified  in  terms  of  (l)  item  number  in  the  set  of  72  items  — this 
set  was  known  as  "Key  T000,"  and  (2)  item  number  by  examination  of  the  Class  VII 


CCS  examinations 
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TABLE  u.l 


LIST  OF  72  REPRESENTATIVE  ITEMS  IN  THE  OCS  CUSS  VIII  EXAMINATIONS 
SUBJECTED  TO  ITERATIVE  FACTOR  ANALYSIS 


Item  No. 

Validity 

Difficulty 

Item  No., 

Dec.  ’52 

Item 

Classification 

Classification 

Key  TO 00 

OCS  Exam 

Type 

r 

D 

ORIENTATION  *UD  MILITARY  JUSTICE 


1 

9 

B 

High 

.25 

Low 

.36 

2 

16 

A 

High 

.37 

High 

.85 

3 

17 

A 

Low 

.06 

High 

.91 

b 

20 

A 

Low 

.0? 

Med. 

.81 

5 

21 

A 

Med. 

.08 

Low 

.55 

6 

2b 

A 

Low 

.08 

Low 

.55 

7 

26 

A 

Med. 

.09 

Med. 

.78 

8 

31 

A 

High 

.25 

Low 

.35 

9 

hh 

B 

Med. 

.10 

High 

.99 

10 

51 

B 

Low 

.00 

Low 

.n2 

11 

61 

A 

High 

.2b 

Med. 

.72 

12 

63 

B 

Med. 

.17 

Low 

.n5 

13 

65 

A 

Med. 

.16 

High 

.89 

m 

69 

B 

Low 

-.05 

High 

.92 

15 

70 

B 

Med. 

.18 

Med. 

.82 

16 

71 

B 

Low 

.01 

Med. 

.6n 

17 

72 

B 

High 

.22 

Med. 

.70 

18 

75 

B 

High 

.21 

High 

.97 

OPERATIONS 

19 

1 

A 

Med. 

.18 

Med. 

.68 

20 

2 

A 

Low 

.11 

High 

.93 

21 

4 

B 

Low 

.07 

Low 

.38 

22 

11 

A 

High 

.Ll 

Med. 

.72 

23 

lb 

A 

Low 

.08 

Mori 

.7n 

2 b 

18 

B 

High 

.-26 

Wi  crh 

pi. 

0 v»l  { 

25 

25 

B 

Med. 

.22 

Med. 

.72 

26 

28 

B 

Low 

.Ob 

Med. 

.73 

27 

29 

A 

Med. 

.18 

High 

.90 

28 

30 

B 

High 

.HO 

Med. 

.52 

29 

31 

B 

High 

.31 

Low 

MS 

30 

36 

A 

Low 

.02 

Low 

MS 

31 

bO 

A 

High 

.36 

Low 

.32 

32 

bh 

A 

High 

.31 

High 

.86 

33 

60 

A 

Med. 

.20 

Low 

.27 

3U 

70 

B 

Med. 

.18 

Low 

.30 

35 

71 

B 

Med. 

.15 

High 

.91 

36 

75 

B 

Low 

.in 

High 

.80 
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Iterative  Factor  Analysis  of  72  Items 


The  Wherry -Gaylord  procedure  was  then  applied  to  these  ? 2 items.  The  data 
studied  were  the  responses  of  a random  sample  of  200  cases  from  Class  VlH.  This 
size  of  sample  was  selected  as  sufficient  in  view  of  the  fact  that  the  procedure 
had  previously  been  used  successfully  on  231  cases  by  Wherry,  Campbell.,  and  Perloff.* 
The  item-test  correlations  used  in  this  procedure  were  exclusively  tetrachoric 
correlations  using  a 50-50  criterion  dichotomy.  A special  table  of  tetrachoric 
correlations  based  on  the  50-50  dichotomy  was  prepared,  in  as  much  as  the  chart 

■ft*# 

published  by  Adkins  could  not  be  read  with  sufficient  precision.  Correlations 
which  could  not  be  read  from  the  Thurstone  tables  were  computed  by  the  cosine 
formula. 

The  procedure  resulted  in  five  factors.  The  final  keys  which  resulted  were 
identified  as  S105,  S207,  S309,  SU01,  and  S501.  The  item-correlations  with  these 
five  keys  are  shown  in  Table  U.2»  which  also  specifies  the  items  included  in  the 
keys.  In  obtaining  scores  on  the  keys,  item  scores  were  added  with  unit  weights. 

The  characteristics  of  the  scores  their  intercorrelations,  and  the  possible 
meaning  of  the  factors  will  be  discussed  at  a 1 ater  point  in  this  report, 
in  conjunction  with  other  results. 


-ft 


Wherry,  R.  J.,  Perloff,  R. , and  Campbell,  J.  T. 
of  the  Wherrv-Craviord  iterative  factor  analysis 

1951,  16,  67-7U7 


An  empirical  verification 
pr  o c o dur  c . Its^rckcme^ujr^jOci  $ 


Page  I8I4.  in  Adkins,  D.  C. , The  Construction  and  Analysis  of  Achievement  Tests. 
Washington,  D.C.,  Government  Printing  Office,  19U7# 


Chesire,  L.,  Saffir,  M.,  and  Thurstone,  L.L.  Computing  Diagrams  for  the 
Tetrachoric  Correlation  Coefficient.  Chicago:  Univ.  Chicago  Bookstore,,  1933* 
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-17 

U5* 

13 

07 

03 

0 

16 

2. 

08 

52* 

ii9* 

-08 

-02 

0 

17 

3. 

51i* 

-05 

3li* 

05 

13 

0 

20 

h. 

UO* 

-12 

03 

12 

22 

0 

21 

5. 

38* 

00 

00 

16 

07 

0 

2h 

6. 

30* 

00 

-02 

-08 

05 
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26 
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12 

12 

09 

00 

3U* 

0 

31 

8. 

23 

13 

07 

li6* 

-03 

0 

hlx 

9. 
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100 

62 

25 

-25 

0 

51 
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08 

05 

-08 

26 

0 

61 
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Hi 

05 

39 

07 

03 

0 

63 
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17 
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07 

0 
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00 

11 
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69 
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13 
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25* 

100* 

0 

70 

15. 

35 

05 

55* 

15 

30 

0 

71 

16. 

-08 
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02 

08 

U2* 

0 
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17. 

5o* 

13 

50* 

13 

03 

0 

75 

18. 

00 

00 

li2 

-U2 

-U2 

0? 

1 

19. 

02 

22 

15 

U5* 

-18 

OP 

A 

c 

20. 

13 

-05 

05 

3ii* 

-13 

OP 

h 

21. 

-03 

02 

07 

ii3* 

38 

OP 

11 

22. 

16 

li2* 

56* 

-06 

16 

OP 

lli 
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05 

12 

16 
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-05 

OP 

18 

2li. 

li2* 

32 
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02 

15 

OP 

25 

25. 

18 
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ii7* 

18 
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OP 

28 

26. 

li6* 

20 

25* 

12 

00 

OP 

29 

27. 

13 

05 

05 

13* 

3li 

OP 

30 

28. 

37 

IiO 

66* 

28 

in 

OP 

31 

29. 

36 

32* 

66* 

03 

-12 

OP 

36 

30, 

03 

-11 
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-lli 

51* 

HD 

W4 

iiO 

31. 

2li 

51* 

31* 

02 

20 

OP 

lili 

32. 

lli 

05 

27 

-08 

5o* 

OP 

60 

33. 

02 

15 

00 

59* 
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OP 

70 
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-05 
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71 
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09 
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00 

OP 
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Denotes 

items  which  formed  the  corresponding  nS"  key. 

*# 

Key: 

0 = Orientation  and  Military  Justice 

OP  = Operations 
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TABLE  U.2  (continued) 


No.  on 
Dec.  OCS 
Exam#* 

.Item  # 
Key  T000 

sio5 

S207 

Factor  Key 
S3  09 

SL01 

S501 

S 

30 

37. 

12 

63* 

51* 

20 

-18 

s 

ill 

38. 

5ii* 

00 

27* 

18 

08 

s 

uu 

39. 

25 

00 

00 

00 

Uo* 

N 

1 

liO, 

00 

66* 

Uo* 

00 

U5 

N 

h 

Ul. 

18 

18 

52* 

02 

-02 

N 

5 

U2. 

06 

Uo 

29 

06 

17 

N 

7 

U3. 

03 

18 

37 

18 

22 

N 

16 

Ui. 

13 

53* 

U3 

17 

-10 

N 

18 

U5. 

02 

lU 

18 

60* 

-2h 

N 

23 

L6. 

00 

15 

13 

13 

62* 

N 

26 

U7. 

25 

-3H 

12 

-18 

3U* 

N 

27 

L8. 

07 

07 

03 

-18 

50* 

N 

37 

U9. 

"05 

25 

3U* 

-05 

13 

N 

H5 

50. 

-52 

36 

17 

3h 

3U 

N 

H6 

51. 

-03 

00 

-07 

lj.2* 

-16 

N 

52 

52. 

12 

12 

52* 

08 

08 

N 

58 

53. 

5o* 

lb 

56* 

-05 

05 

N 

60 

5U. 

06 

08 

uU* 

-ou 

-11 

W 

3 

55. 

23 

39* 

36* 

07 

20 

W 

h 

56. 

02 

-10 

03 

02 

28* 

w 

5 

57. 

L2 

-JU2 

00 

100 

-U2 

w 

11 

58. 

35* 

03 

12 

12 

16 

w 

15 

59. 

03 

07 

07 

18* 

00 

w 

16 

60. 

10 

07 

07 

52* 

-03 

w 

17 

61. 

67* 

09 

16 

09 

2L 

w 

27 

62. 

38* 

26 

h5* 

-38 

-Ok 

w 

31 

63. 

1U 

1U 

35* 

00 

05 

w 

32 

6Uo 

-07 

-10 

16 

31* 

02 

w 

38 

65. 

-13 

L7* 

16 

07 

07 

w 

39 

66. 

00 

23 

26 

30* 

-07 

w 

U9 

67. 

21 

6u* 

37 

18 

-06 

w 

53 

68. 

16 

16 

16 

-07 

56* 

w 

55 

69. 

-06 

02 

13 

U6* 

12 

w 

£7 

7 n 

OR 

iil 

0*7 

■3  O 
J-L 

7 rv 
”1U 

w 

68 

71. 

05 

-05 

-39 

-10 

61 

w 

69 

72. 

02 

50* 

U 

02 

••1U 

Denotes  items  which  formed  the  corresponding  nSn  key. 

*-* 

Keys  S = Seamanship 
N = Navigation 
W = Naval  Weapons 
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Expansion  of  the  Factor  Keys 


Since  the  S-keys  contained  relatively  few  items,  the  scores  were  then  used  as 
criteria  for  identifying  other  items  among  the  ifLO  items  available  which  could 
be  considered  as  meas_'’ing  the  same  factors.  The  same  sample  of  200  cases  was 
used  as  for  the  previous  analysis.  The  item-key  correlations,  again  computed  by 
the  special  tetrachoric  correlation  tables  involving  a 50-50  dichotomy,  are  shown 
in  those  columns  of  Table  iu3  which  are  headed  by  the  number  of  one  of  the  S-keys, 

In  the  case  of  items  which  appeared  on  S-keys,  the  item-key  correlations  may  not 
be  the  same  as  those  shown  in  Table  kc2  because  of  the  fact  that  the  upper  and 
lower  SO  per  cent  groups  on  the  criterion  groups  were  sorted  in  a slightly  different 
way.* 

Items  were  then  selected  for  the  expanded  keys  by  the  following  rules: 

(a)  All  items  on  the  original  S-keys  were  retained,  except  as  provided  by 
rule  (c)  below. 

(b)  New  items  must  have  a correlation  of  .30  or  greater  with  the  S-key, 

(c)  The  item  could  be  assigned  to  no  more  than  one  E-key,  that  is,  to  the 
factor  where  its  item-key  correlation  was  highest. 

(d)  Very  easy  items  showing  unstable  item-key  correlations  were  to  be  ignored. 

The  results  of  this  "Operation  Dragnet"  were  disappointing.  Relatively  few 

new  items  were  added  to  the  S-keys  to  form  the  E-keys.  In  fact,  all  five  E-keys 
incD'ided  only  136,  or  33  per  cent  of  the  UlO  items.  Perhaps  the  requirement  that 
the  correlations • had  to  be  ,30  or  greater  was  too  rigorous.  Correlations  of  „30 
or  greater  were  relatively  infrequent  in  the  data.  But  it  must  be  remembered  that 
an  attempt  was  being  made  to  identify  separate  aspects  of  achievement;  hence  it 
was  necessary  to  be  assured  of  clarity  of  the  results  in  order  to  make  interpreta+icns. 
It  had  been  thought  that  many  new  items  in  the  total  of  IfLO  available  would  fall  into 
one  of  the  E-keys, 

* In  every  case,  the  papers  having  a score  at  the  median  were  randomly  assigned  to 
the  upper  and  lower  groups;  this  random  assignment  was  usually  different  each 
time  the  score  was  used  as  a criterion. 
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TABLE  k.3 

CORRELATIONS  OF  ALL  ITEMS 

IN  DECEMBER  OCS  EXAMINATIONS  WITH  "S"  KEYS  AND  "E"  KEYS 

Note.  # Denotes  that  item  was  selected  for  the  corresponding  E Key. 
**  Denotes  that  item  was  in  the  original  "S"  Key. 

***  Denotes  item  on  S Key  and  on  E Key. 

All  entries  have  been  multiplied  by  100  to  eliminate  decimal  point. 


Item  No, 

S105 

E100 

S207 

E200 

S309 

E300 

Skoi 

EkOO 

S501 

ORIENTATION  EXAMINATION 

1. 

05 

22 

05 

-05 

02 

02 

05 

2k 

12 

2. 

10 

10 

-10- 

-10 

-10 

-10 

-10 

10 

-10 

3. 

17 

-12 

-12 

-08 

-05 

-16 

02 

00 

12 

10 

00 

27 

27 

31* 

39 

00 

-05 

-05 

5. 

20 

10 

13 

17 

17 

13 

00 

08 

-07 

6. 

20 

30 

-05 

09 

16 

12 

30* 

59 

12 

7. 

00 

100 

100 

-100 

-100 

-100 

00 

00 

00 

8. 

20 

ok 

12 

35 

ok 

20 

60* 

73 

-ok 

9. 

-13 

-10 

k5**i 

* 33 

03 

10 

13 

17 

-03 

10. 

-27 

-21 

-20 

-21 

-27 

-27 

-21 

-21 

21 

11. 

22 

-13 

02 

10 

18 

06 

13 

11 

06 

12. 

16 

08 

08 

26 

08 

20 

08 

16 

-15 

13. 

08 

08 

02 

-08 

18 

13 

18 

18 

02 

lk. 

00 

15 

00 

-05 

10 

05 

-10 

15 

-05 

15. 

ok 

ok 

16 

10 

30* 

16 

-2k 

”10 

ok 

16. 

08 

22 

)|]|»* 

33 

k3*** 

k8 

02 

08 

02 

17. 

kO***  38 

o5 

00 

3k**  ' 

12 

05 

-13 

25 

18. 

10 

17 

13 

03 

r\i 
^ 1 

“07 

-07 

02 

00 

19. 

10 

10 

23 

17 

23 

10 

03 

13  • 

07 

20. 

30**-*  ko 

-15 

-07 

-08 

02 

08 

-08 

25 

21. 

33***  36 

03 

00 

03 

03 

07 

16 

07 

22. 

15 

28 

21 

12 

2k 

2k 

-05 

05 

12 

23. 

“09 

17 

00 

-09 

-05 
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00 

00 

30* 

2k. 

26** 

05 

00 
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05 

05 

-12 

-lk 

00 

25. 
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-10 

15 

18 

03 

13 

13 

03 

15 
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12 

12 

12 

00 

05 
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00 
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3k*** 
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100 
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10 

ok 
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22 
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k8 

10 

-16 

16 
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08 

-08 
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07 
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31. 

20 

27 

17 
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22 
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18 
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05 

02 

09 

20 

05 
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13 

22 
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07 

00 

07 

07 

08 

16 
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TABLE  lx. 3 (continued) 
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TABLE  1+.3  (continued) 


Item  No, 
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E300 
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35. 

-05 

05 

12 

18 

12 

12 

05 

12 

05 

36. 
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39. 

18 

21+ 

22 

10 

22 

18 

-03 

12 

03 

HO. 

09 

16 

1x8*** 

1+8 

Ip.** 

28 

05 

05 

28 

ill. 

100 

100 

-100 

— 

100 

— 

100 

100 

100 

1+2. 

"16 

00 

00 

-03 

03 

r\  -» 

m 

_<V7 
^ 1 

n7 

— 1 

1+3. 

15 

18 

08 

08 

-08 

15 

02 

15 

-12 

1+1+. 

16 

20 

16 

08 

26 

30 

02 

-02 

1x8*** 

1+5. 

-05 

H+ 

10 

05 

19 

29 

11+ 

36 

10 

1+6. 

27 

27 

08 

08 

H+ 

18 

08 

03 

18 

1+7, 

07 

07 

07 

03 

-03 

00 

18 

15 

07 

1+8. 

09 

09 

18 

22 

13 

26 

-13 

18 

05 

1+9. 

07 

00 

15 

19 

15 

30 

22 

08 

-03 

50. 

00 

19 

07 

10 

19 

19 

H+ 

10 

-10 

51. 

07 

10 

-03 

03 

10 

07 

07 

07 

-23 

52. 

11+ 

-02 

11+ 

26 

06 

10 

lix 

00 

06 
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TABLE  4.3  (continued) 


Item  No. 

S105 

E100 

S207 

E200 

S309 

E300 

s4oi 

E400 

s5oi 

53. 

02 

-08 

20 

12 

24 

14 

08 

14 

-05 

54. 

21 

31 

-02 

12 

24 

18 

34* 

42 

15 

55. 

-15 

-12 

-up 

02 

02 

02 

-02 

10 

02 

56. 

-12 

-12 

12 

00 

07 

-12 

20 

26 

00 

57. 

17 

17 

34 

34 

34 

34 

-17 

17 

34 

58. 

00 

07 

07 

"07 

16 

25 

03 

00 

-10 

59. 

-18 

-13 

22 

28 

-03 

07 

10 

26 

-22 

6o. 

-05 

10 

12 

12 

-08 

-05 

49*** 

52 

-26 

61. 

00 

22 

05 

08 

-15 

-05 

05 

02 

-19 

62. 

05 

10 

16 

27 

10 

10 

05 

16 

-05 

63. 

02 

-02 

22 

32 

-02 

02 

22 

40 

•05 

64. 

20 

16 

17 

30 

05 

02 

11 

10 

-02 

65. 

05 

34 

-05 

00 

-30 

-24 

-24 

oc 

-24 

66. 

16 

20 

18 

— o4 

-06 

-04 

-33 

-37 

04 

67. 

45 

30 

10 

o4 

10 

04 

45 

24 

30 

68. 

-13 

13 

13 

25 

-05 

-05 

05 

05 

13 

69. 

34* 

34 

13 

08 

08 

-12 

00 

08 

08 

70. 

47*** 

60 

-05 

16 

05 

02 

08 

08 

19 

71. 

ho*** 

69 

18 

18 

00 

oc 

-11 

11 

11 

72. 

27 

36 

18 

11 

00 

-11 

11 

27 

00 

73. 

23 

2j 

26 

23 

20 

30 

10 

23 

03 

74. 

18 

11 

14 

32 

18 

18 

-02 

14 

-07 

75. 

33*** 

19 

-14 

05 

10 

15 

-10 

15 

00 

SEAMANSHIP  EXAMINATION 

1. 

-42 

25 

25 

42 

-25 

00 

42 

42 

00 

2. 

00 

00 

00 

-21 

42 

21 

-66 

00 

-42 

3. 

03 

18 

16 

10 

00 

13 

22 

28 

18 

4. 

23 

14 

26 

28 

39* 

35 

08 

08 

*02 

5. 

03 

-03 

07 

03 

0? 

00 

07 

18 

-03 

6. 

08 

18 

18 

02 

24 

24 

02 

08 

13 

7. 

10 

"03 

18 

31 

22 

13 

07 

13 

00 

8. 

09 

02 

-16 

-12 

02 

-05 

20 

09 

-05 

9. 

-40 

-40 

12 

07 

07 

12 

07 

26 

-12 

10. 

09 

09 

05 

20 

-09 

11 

18 

34 

-14 

11. 

-25 

25 

24 

-62 

-24 

-24 

-13 

24 

-25 

12. 

27 

30 

11 

- A 

XO 

14 

r\rf 

~c\C 

]_T_ 

ns 
— >» 

13. 

29 

12 

19 

05 

32* 

38 

05 

13 

02 

14. 

-06 

-02 

22 

06 

10 

06 

-13 

•02 

02 

15. 

12 

15 

09 

18 

15 

05 

—02 

07 

12 

16. 

-16 

-05 

-05 

-12 

02 

-12 

12 

24 

-12 

17. 

16 

34 

-04 

-10 

04 

10 

26 

34 

-04 

18. 

08 

20 

34* 

32 

20 

34 

00 

00 

-13 

19. 

22 

18 

28 

18 

18 

22 

10 

14 

-14 

20. 

16 

10 

16 

27 

10 

00 

16 

16 

00 
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TABLE  U.3  (continued) 


Item  No, 

sio5 

E100 

S207 

E200 

S309 

E300 

SU01 

EU00 

S501 

21. 

27 

05 

10 

16 

10 

00 

-10 

-05 

-10 

22. 

“16 

-18 

12 

03 

-03 

00 

10 

12 

07 

23. 

21* 

30 

2U 

00 

12 

00 

-12 

00 

00 

2U. 

03 

08 

08 

08 

lli 

08 

08 

-03 

08 

25. 

18 

13 

18 

2li 

18 

2U 

-08 

13 

-2U 

26. 

3U* 

52 

29 

25 

25 

20 

07 

11 

02 

27. 

13 

02 

21* 

28 

18 

18 

18 

-02 

28 

28. 

05 

-05 

2li 

2li 

29 

29 

05 

00 

-10 

29. 

10 

25 

03 

07 

07 

10 

13 

10 

07 

30. 

18 

11* 

62*** 

62 

5U** 

U5 

12 

20 

-20 

31. 

12 

18 

11 

12 

-10 

03 

18 

12 

00 

32. 

13 

2U 

10 

08 

28 

2U 

13 

-02 

12 

33. 

100 

100 

00 

00 

00 

00 

00 

00 

100 

3U. 

16 

19 

07 

10 

03 

16 

22 

22 

-■19 

35. 

o5 

15 

-10 

-12 

-15 

08 

00 

08 

-15 

36. 

13 

10 

13 

13 

13 

00 

-12 

00 

08 

37. 

“13 

-07 

10 

-10 

00 

13 

12 

13 

00 

38. 

25 

o5 

25 

25 

-13 

-05 

25 

38 

-13 

39. 

-o5 

-02 

-09 

-20 

-02 

-lli 

-05 

-05 

08 

Uo. 

-07 

10 

33* 

31 

31 

22 

03 

22 

-10 

Ui. 

5o** 

55 

18 

27 

36** 

18 

00 

18 

18 

U2. 

16 

22 

18 

10 

28 

31 

00 

-28 

25 

U3. 

09 

02 

19 

09 

05 

05 

12 

05 

02 

Ut. 

lli 

28 

-lli 

-lli 

00 

00 

1U 

-lli 

U3*** 

1*5. 

-08 

03 

18 

03 

03 

18 

-02 

-08 

13 

U6. 

09 

23 

02 

09 

09 

12 

02 

02 

02 

U7. 

09 

o5 

2k 

20 

15 

17 

-02 

05 

02 

1*8. 

12 

-22 

02 

02 

09 

05 

-08 

09 

-02 

1*9- 

05 

15 

15 

12 

15 

-02 

05 

08 

-02 

5o. 

10 

10 

16 

16 

10 

10 

OU 

16 

-22 

NAVIGATION 

EXAMINATION 

i 

00 

-11 

5o  ** 

55 

36** 

27 

11 

-11 

55 

2. 

o5 

-o5 

12 

02 

02 

12 

12 

12 

08 

3. 

07 

12 

30 

Uo 

53* 

65 

12 

12 

00 

4. 

26 

lli 

11 

22 

45*** 

44 

-02 

02 

02 

5. 

15 

08 

38* 

Uo 

22 

20 

06 

-OU 

20 

6, 

-09 

07 

12 

12 

07 

07 

18 

15 

05 

7. 

00 

10 

20 

2U 

31* 

34 

13 

10 

22 

8. 

11 

22 

-11 

-11 

-11 

11 

00 

00 

-22 

9. 

22 

22 

22 

30 

UO 

Uo 

uo 

22 

-11 

10. 

07 

10 

22 

18 

22 

31 

16 

22 

03 

11. 

18 

2k 

15 

20 

05 

15 

15 

12 

05 

12. 

03 

00 

-03 

-03 

07 

-03 

-07 

-19 

13 

13. 

08 

02 

15 

26 

22 

26 

15 

26 

08 

11*. 

27 

27 

-21 

-21 

100 

27 

27 

21 

21 

15. 

10 

-03 

36* 

U2 

20 

18 

iu 

2U 

-Hi 

TABLE  1*.3  (continued) 


Item  No* 

sio5 

E100 

S207 

E200 

S309 

E300 

■ Sl*Gl 

Ei*00 

s5oi 

16. 

02 

13 

57*** 

60 

1*3 

1*2 

21* 

29 

-07 

17. 

16 

1*0 

30* 

30 

00 

-01* 

09 

-01* 

00 

18. 

02 

H* 

23 

21* 

18 

12 

58*** 

51 

-16 

19. 

-100 

-100 

100 

100 

-100 

100 

-100  - 

■100 

100 

>0. 

05 

17 

1*2* 

1*2 

20 

12 

12 

36 

-08 

21. 

-05 

-25 

05 

25 

00 

05 

05 

13 

-05 

22. 

13 

20 

05 

20 

20 

13 

-26 

-05 

12 

23. 

21* 

36 

21* 

-21* 

-21* 

-21* 

11* 

21* 

100** 

2l*. 

30* 

1*0 

09 

09 

09 

21* 

-09 

-09 

-16 

25. 

10 

-03 

33* 

37 

H* 

26 

11* 

18 

-03 

26. 

25 

12 

-25 

-12 

05 

18 

-18 

05 

1*8*** 

27. 

00 

07 

07 

-03 

11* 

03 

-18 

-30 

1*3* 

28. 

29 

18 

H* 

03 

29 

32 

10 

H* 

-03 

29. 

18 

18 

28 

31* 

31* 

1*2 

08 

18 

-05 

30. 

00 

21* 

07 

13 

-03 

03 

10 

03 

22 

31. 

16 

16 

21* 

05 

32 

32 

-21* 

-02 

58* 

32. 

07 

00 

18 

22 

31** 

31* 

-05 

16 

-07 

33. 

-1*2 

-1*2 

1*2 

00 

00 

00 

1*2 

1*2 

-100 

31*. 

58* 

21 

27 

5o 

27 

58 

— 

100 

-100 

35. 

00 

-27 

27 

00 

00 

27 

-27 

27 

00 

36. 

— 

-- 

— 

— „ 

— 

— 

— 

— 

37. 

-05 

-05 

13 

05 

3l*** 

25 

05 

-13 

31* 

38. 

-02 

-07 

02 

18 

15 

25 

02 

08 

-05 

39. 

00 

00 

00 

00 

00 

00 

00 

00 

00 

1*0. 

-21 

21 

-21 

-100 

-21 

-21 

21 

-21 

27 

ia. 

27 

-27 

25 

11* 

25 

25 

-11* 

27 

H* 

1*2, 

-07 

-31* 

20 

31* 

3li* 

31* 

07 

07 

-07 

1*3. 

17 

17 

17 

17 

00 

52 

17 

17 

-31* 

1*1*. 

-27 

-21 

27 

-21 

21 

21 

21 

21 

27 

1*5. 

-52 

“31* 

31* 

31* 

17 

00 

17 

52 

17 

1*6. 

nn 

10 

-10 

-03 

-07 

-03 

38*** 

50 

-10 

1*7. 

-5o 

-5o 

-10 

-10 

10 

-10 

10 

30 

-30 

1*8. 

21 

21 

21 

-21 

21 

-21 

27 

-21 

27 

1*9. 

ol* 

01* 

01* 

26 

16 

10 

01* 

10 

Ol* 

5o. 

07 

-02 

07 

11 

16 

20 

20 

11 

20 

51. 

09 

12 

12 

09 

20 

16 

-05 

05 

09 

52. 

08 

02 

08 

17 

52*** 

52 

02 

12 

02 

uj. 

JO* 

33 

26 

o 

38 

uo 

n r\ 

J .c 

51*. 

21* 

21* 

1*0* 

— 

30 

1*0 

11* 

21* 

10 

55. 

00 

07 

-08 

-03 

08 

11 

31* 

31 

-03 

56. 

17 

13 

16 

00 

31** 

30 

03 

-03 

00 

57. 

07 

07 

07 

11* 

u* 

18 

11* 

21 

00 

58. 

5o** 

20 

08 

-05 

53*** 

53 

-17 

-05 

-05 

59. 

26 

02 

11* 

20 

1*8* 

51* 

18 

18 

05 

60. 

11 

-11 

-Ol* 

08 

)|)|»*» 

1*1* 

-11 

-11 

-16 

i 

k 

i — 66  ~ 


i 


TABLE  L.3  (continued) 


i 


Item  lie 

1.  3105 

E100 

S207 

E200 

S309 

E300 

SL01 

ELOO 

S501 

E1001 

WEAPONS  EXAMINATION 

1. 

16 

26 

26 

20 

20 

26 

57* 

51 

-08 

-08 

2. 

08 

03 

Hi 

19 

1L 

19 

03 

-03 

-06 

00 

3* 

20 

17 

39«-a- 

36 

39** 

30 

23 

17 

20 

08 

k. 

02 

02 

-06 

-02 

-02 

22 

02 

LO***  28 

5. 

00  • 

1+2 

-li2 

00 

L2 

L2 

100 

L2 

00 

100 

6. 

00 

00 

00 

00 

00 

00 

00 

00 

00 

36 

7« 

1+5 

20 

20 

20 

50* 

60 

— Oil 

-20 

OL 

11 

8. 

32* 

60 

-Oo 

02 

15 

02 

18 

18 

05 

38 

9. 

-03 

16 

03 

-03 

10 

10 

26 

22 

03 

2L 

10. 

00 

00 

00 

00 

58* 

56 

00 

00 

00 

-10 

11. 

LO***  1+7 

03 

12 

07 

16 

16 

1L 

18 

38 

12. 

02 

-07 

36 

32 

36 

36 

05 

1L 

-05 

13 

13. 

18 

2li 

2li 

18  . 

22 

13 

-03 

10 

22 

18 

1 L. 

10 

16 

22 

12 

12 

10 

22 

2L 

03 

16 

15. 

10 

10 

07 

Hi 

-18 

-05 

37***  33 

-03 

05 

16. 

16 

19 

03 

00 

07 

13 

60***  60 

03 

20 

17. 

50***  30 

09 

30 

2L 

30 

00 

09 

2L 

L8 

18. 

12 

-07 

-03 

00 

00 

00 

03 

07 

-18 

07 

19. 

18 

10 

22 

16 

16 

16 

07 

10 

00 

15 

20. 

00 

00 

25* 

50 

00 

00 

27 

00 

-27 

-36 

21. 

Hi 

02 

Hi 

23 

26 

23 

11 

-02 

02 

18 

22. 

16 

16 

-02 

-03 

13 

03 

00 

12 

-05 

05 

23. 

00 

13 

28 

28 

16 

10 

2L 

13 

07 

18 

2U. 

07 

20 

03 

-10 

07 

03 

07 

10 

03 

-02 

25. 

03 

12 

29 

18 

21 

10 

18 

10 

21 

15 

26. 

09 

30 

-16 

-17 

-09 

-21 

00 

-22 

16 

22 

27. 

1+5** 

38 

26 

15 

52***  L5 

-20 

-26 

-03 

06 

28. 

05 

22 

00 

10 

05 

02 

00 

-05 

05 

02 

29. 

— 

00 

— 

— 

— 

— 

00 

— 

— 

— 

30. 

05 

20 

16 

2li 

16 

23 

03 

08 

03 

2L 

31. 

lii 

17 

Hi 

28 

30***  L6 

02 

12 

08 

02 

32. 

00 

00 

-03 

-13 

-13 

-08 

25** 

-03 

03 

02 

33. 

15 

20 

-07 

-08 

12 

12 

17 

12 

20 

16 

3U. 

-05 

05 

o5 

11 

02 

10 

2L 

20 

02 

05 

06 

06 

00 

06 

17 

17 

06 

-06 

00 

OL 

36. 

uii 

26 

16 

CO 

f'S 

1 ! 
44 

Hi 

r\r\ 

£.\J 

~i  c. 

±\J 

AC? 

1 A 

XV 

37. 

21 

28 

28 

Li 

32* 

50 

10 

10 

00 

30 

38. 

-13 

-16 

ii7***  38 

13 

16 

19 

19 

00 

05 

39. 

07 

10  ' 

20 

17 

20 

23 

32***  36 

-05 

-02 

Lo. 

Hi 

18 

10 

18 

10 

18 

10 

10 

06 

12 

Cross  Validation  Sample 
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TABLE  4.3  (continued) 


Item  No. 

3105 

E100 

3207 

E200 

S309 

E300 

SltOl 

E400 

s5oi 

E100 

111. 

22 

22 

29 

22 

29 

1*0 

00 

17 

00 

10 

k 2. 

“05 

-08 

12 

22 

22 

18 

-08 

02 

05 

12 

1*3. 

07 

09 

18 

22 

35* 

1*1* 

05 

05 

02 

09 

ill*. 

23 

16 

16 

10 

23 

26 

16 

22 

22 

24 

U5. 

-07 

-12 

07 

20 

20 

26 

00 

00 

-12 

08 

1*6. 

09 

06 

16 

16 

12 

21* 

-02 

-02 

08 

-04 

1*7. 

21* 

11* 

33* 

1*2 

26 

38 

18 

lit 

-02 

02 

1*8. 

17 

14 

11 

08 

-02 

05 

11 

22 

08 

02 

1*9 » 

10 

18 

68*** 

58 

32 

37 

18 

18 

-06 

16 

50* 

-05 

-05 

52* 

52 

21* 

29 

05 

05 

-10 

38 

51* - 

00 

-08 

37* 

1*5 

33 

37 

12 

00 

04 

28 

52. 

22 

10 

31** 

1*5 

30 

1*7 

-10 

06 

10 

33 

53* 

13 

07 

22 

13 

16 

19 

10 

-07 

62*** 

03 

51*. 

13 

17 

20 

27 

20 

27 

20 

20 

-20 

07 

55* 

<-02 

-02 

-02 

-06 

16 

06 

46*** 

32 

02 

18 

56. 

"01* 

11 

11 

36 

08 

08 

20 

15 

00 

20 

57* 

'25 

18 

37 

1*1* 

37 

1*1 

25 

41 

06 

24 

58.’ 

22 

13 

29 

38 

29 

32 

07 

22 

00 

03 

59. 

15 

15 

30* 

20 

21* 

38 

10 

05 

05 

17 

60* 

09 

09 

15 

15 

22 

25 

-12 

-03 

-05 

18 

61* 

05 

08 

05 

08 

08 

-03 

-03 

07 

-12 

13 

62. 

25 

25 

03 

19 

-03 

03 

-03 

25 

-08 

40 

63. 

01* 

01* 

— 

-01* 

01* 

•••• 

•aaa 

-02 

61*. 

00 

05 

26 

20 

32* 

38 

00 

02 

26 

22 

65. 

07 

00 

03 

07 

10 

—03 

07 

03 

-07 

-03 

66. 

30 

35 

35 

38 

25 

25 

35 

32 

-05 

02 

67. 

00 

07 

11 

11 

11 

23 

07 

23 

07 

06 

68. 

21* 

08 

08 

1*0 

-1*0 

21* 

21* 

-08 

-24 

20 

69. 

-10 

02 

50*** 

5o 

10 

11* 

-02 

06 

02 

30 

70. 

10 

lit 

20 

18 

18 

20 

-07 

07 

03 

14 

71. 

09 

00 

23 

23 

•05 

19 

00 

05 

05 

14 

72. 

-10 

-20 

20 

30 

-07 

07 

-07 

-07 

00 

20 

73. 

19 

-05 

13 

09 

23 

27 

-10 

-13 

-09 

09 

71*. 

16 

10 

31 

22 

31 

1*0 

-03 

03 

13 

22 

75. 

10 

10 

10 

21* 

20 

15 

10 

15 

00 

00 

ENGINEERING 

EXAMINATION 

— « 

13 

16 

03 

00 

07 

07 

34* 

38 

00 

15 

2. 

-03 

00 

-07 

-12 

03 

-03 

-22 

-26 

-03 

-07 

3. 

31* 

28 

28 

1*0 

36* 

36 

12 

05 

08 

20 

1*. 

16 

22 

23 

36 

36* 

1*1* 

23 

23 

12 

24 

5. 

-08 

08 

08 

08 

17 

08 

17 

30 

02 

05 

6. 

18 

22 

30 

1*3 

37* 

33 

0? 

16 

30 

-02 

7. 

-13 

-10 

-13 

-lit 

-07 

-10 

-10 

-22 

‘ -10 

10 

8. 

17 

17 

07 

-03 

13 

17 

13 

03 

20 

07 

9. 

06 

06 

-06 

00 

-06 

-18 

13 

09 

02 

14 

10. 

16 

08 

16 

32 

28 

32 

-04 

-02 

-06 

19 
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TABLE  U.3  (continued) 


Item  No. 

sio5 

E100 

S207 

E200 

S309 

E300 

SiiOl 

EiiOO 

S501 

E100 

11. 

18 

18 

22 

33 

22 

18 

33* 

52 

-06 

08 

12. 

07 

07 

10 

10 

22 

13 

18 

15 

03 

lii 

13. 

28 

16 

ii5* 

62 

i|2 

52 

2ii 

32 

-06 

11 

111. 

20 

30 

32 

i|2 

32 

2i| 

18 

18 

12 

-02 

15. 

lii 

111 

30* 

30 

26 

lii 

-10 

-10 

06 

-08 

16. 

111 

10 

07 

-03 

13 

13 

-22 

-22 

03 

07 

17. 

Ik 

12 

26 

2ii 

26 

28 

2ii 

08 

30* 

07 

18. 

12 

06 

iiii* 

Ui 

32 

28 

08 

02 

08 

02 

19. 

28 

28 

kh 

36 

iiii 

lii 

-10 

-02 

25 

21 

20. 

29 

32 

12 

18 

22 

32 

-02 

-02 

02 

12 

21. 

-18 

-05 

-05 

“05 

-16 

-12 

-o5 

02 

08 

38 

22. 

39 

10 

28 

39 

39 

28 

03 

07 

07 

-08 

23. 

00 

-09 

2i| 

09 

09 

-06 

Oii 

09 

16 

30 

2i|. 

33 

38 

ii5* 

62 

38 

i|2 

18 

22 

-10 

33 

25. 

08 

02 

35 

32 

i|2* 

38 

12 

26 

-05 

13 

26. 

28 

03 

10 

16 

16 

13 

03 

13 

-10 

-07 

27. 

3.0 

00 

03 

-16 

07 

00 

13 

-03 

16 

17 

28. 

10 

07 

19 

iiO 

25 

3ii 

03 

13 

10 

10 

29. 

iiO* 

52 

08 

17 

29 

22 

22 

17 

33 

3ii 

30. 

26 

20 

26 

32 

26 

20 

111 

20 

08 

19 

31. 

20 

16 

3k 

ii5 

ii5* 

1|2 

28 

2ii 

-02 

05 

32. 

-02 

02 

08 

o5 

05 

05 

12 

02 

02 

13 

33. 

19 

06 

02 

12 

06 

10 

-15 

-06 

-32 

13 

3k. 

25 

25 

16 

25 

29 

20 

12 

08 

Oii 

11 

35. 

16 

08 

35 

35 

2ii 

16 

35 

08 

36 

18 

36. 

2ii 

20 

30* 

56 

20 

20 

18 

20 

lii 

02 

37. 

13 

13 

3ii* 

ii8 

20 

30 

27 

30 

-03 

31 

38. 

07 

10 

-0? 

-03 

-16 

13 

10 

16 

3.3 

-07 

39. 

06 

10 

15 

17 

19 

10 

06 

19 

-06 

06 

1|0. 

00 

03 

10 

20 

27 

23 

10 

07 

-10 

25 

ill. 

22 

30 

30 

71 

50* 

70 

06 

06 

-06 

36 

i|2. 

17 

13 

03 

10 

13 

03 

10 

07 

-07 

28 

U3. 

00 

08 

-07 

-07 

17 

08 

00 

07 

20 

-03 

iiii. 

16 

22 

19 

35 

22 

22 

-07 

0? 

03 

ill 

ii5. 

16 

21 

06 

"03 

12 

08 

08 

08 

-02 

23 

\.c 

t+w* 

*1  1 
-l-J 

1 1 O 

IQ 

-*-/ 

— ^ 

no 
— ✓ 

2? 

o5 

no 
— ✓ 

1*7. 

U2 

i|2 

00 

100 

100 

i|2 

00 

00 

-ICO 

100 

ii8. 

-02 

11 

07 

11 

-02 

07 

16 

2ii 

16 

38 

Ii9. 

111 

2k 

07 

03 

20 

20 

10 

10 

-02 

-15 

50. 

08 

00 

35* 

30 

20 

23 

20 

23 

-11 

12 

51.  35*  i|0  22  05  32  25  02  27  05  1|7 

52.  18  15  15  30  15  20  08  2k  -02  2 3 

53.  16  23  38*  50  35  30  16  35  -16  17 

5ii.  25  18  12  18  25  18  07  10  -16  -13 

55.  Ok  26  15  U5  20  20  27  15  -Ou  -Oil 
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TABLE  4.3  (continued) 


Item  No. 

S105 

El  00 

S207 

E200 

S3  09 

E300 

s4oi 

E400 

S501 

E1T 

56. 

32* 

49 

07 

10 

18 

14 

-03 

-03 

14 

16 

57. 

11 

08 

10 

15 

28 

30 

20 

18 

06 

0? 

58. 

11 

15 

22 

17 

46* 

52 

29 

33 

-29 

08 

59. 

-06 

06 

06 

02 

38* 

34 

-06 

-06 

18 

22 

60. 

06 

11 

02 

07 

06 

02 

22 

-06 

15 

20 

61. 

03 

10 

05 

02 

17 

10 

17 

20 

-07 

-02 

62. 

22 

12 

02 

15 

12 

15 

02 

12 

-12 

-18 

63. 

04 

20 

-20 

09 

-20 

-20 

20 

12 

-04 

25 

64. 

-02 

-02 

02 

05 

05 

05 

12 

12 

-20 

00 

65. 

00 

00 

00 

00 

00 

00 

00 

00 

00 

100 

66. 

22 

28 

-22 

-li 

07 

02 

11 

-07 

07 

22 

67. 

17 

10 

10 

14 

28 

17 

10 

17 

03 

18 

68. 

11 

22 

22 

40 

30* 

22 

00 

-11 

06 

17 

69. 

12 

24 

12 

30 

24 

12 

4o* 

49 

-12 

11 

70. 

42 

21 

21 

21 

21 

21 

-21 

-42 

66* 

08 

71. 

-11 

10 

10 

34 

18 

25 

22 

22 

-08 

-02 

72. 

08 

24 

13 

13 

28 

28 

13 

13 

13 

08 

73. 

08 

08 

38* 

38 

29 

18 

12 

12 

08 

08 

74. 

02 

07 

-16 

-11 

-02 

-11 

02 

02 

16 

02 

75. 

08 

o5 

12 

16 

18 

16 

18 

12 

08 

12 
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At  this  point,  further  investigation  of  Keys  S501  and  E500  was  abandoned, 
since  these  keys  appeared  to  represent  a residual  factor.  The  reliabilities  of 
scores  on  these  keys  were  quite  low,  as  will  be  shown  later. 

An  analysis  of  the  "overall"  validities  of  the  items  on  the  E-keys  (i.e., 
the  validities  against  the  composite  score  criterion  described  in  Chapter  3) 
disclosed,  however,  that  60  (i+9* 6j6)  of  the  121  items  on  Keys  E100,  E200,  F300, 
and  EUOO  together  included  57  per  cent  of  the  high  validity  items  (r  * .30  or 
greater)  on  the  examination  as  a whole.  The  data  of  this  analysis  are  shown  in 
Table  1+.4.  It  is  to  be  noted  that  Keys  E200  and  E300  most  frequently  contained 
items  with  high  overall  validity}  Keys  E100  and  EiiOO  sampled  from  all  ranges  of 
overall  item  validity.  It  may  be  concluded  that  the  dragnet  operation  of  expanding 
the  S-keys  "caught"  a large  number  of  the  available  "good"items,  and  that  at  the 
same  time  some  success  had  been  achieved  in  identifying  (at  least  statistically) 
aspects  of  achievement  not  represented  very  well,  or  masked,  by  the  composite 
criterion.  It  is  possible,  however,  that  some  of  the  items  not  caught  by  the  drag- 
net may  be  measuring  aspects  of  achievement  which  are  not  measured  by  the  "E"  keys. 
This  possibility  could  have  been  investigated  by  Wherry-Gaylord  iterative  procedures, 
but  time  did  not  permit  this  project  to  do  so. 

The  results  were  subjected  to  two  types  of  checking  procedures: 

(l)  Since  the  S-keys  were  not  very  reliable  (as  will  be  shown  below), 

with  a consequent  attenuation  of  item-key  correlations,  the  correlations 
of  all  items  were  computed  against  the  E-keys,  This  was  done  for  all 
E-keys  except  E500,  which  by  this  time  the  project  staff  was  convinced 
was  a residual  factor  not  worthy  of  further  investigation.  The  resulting 
correlations  are  shown  in  Table  U.3.  In  the  main,  it  may  be  concluded 
from  this  table  that  the  correlations  of  the  items  with  the  S-keys 
held  up  well  when  computed  with  the  E-keys, 
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TABLE  k.k 

FREQUENCY  DISTRIBUTION  - (OVERALL)  VALIDITY  COEFFICIENTS 


Dec,  * 52>  0C3 
Exam 
r 

E100 

E200 

E300 

Eaoo 

All 

Other 

Items 

Total 

.55-.59 

•-  - 

2 

2 

.5o-.5a 

2 

1 

1 - 

a 

eli5-.Ji9 

2 

a 

a 

10 

.lio-.aa 

1 

a 

7 

i 

2 

15 

,35-, 39 

1 

10 

10 

3 

16 

ao 

.3Q-..3U 

a 

6 

3 

21 

3a 

,25-, 29 

3 

a 

7 

1 

as 

63 

,20~, 24 

2 

5 

3 

a 

a9 

63 

.15-. 19 

a 

i 

5 

5 

ai 

56 

ilQrylk  • 

2 

2 

3 

35 

a2 

.05-.09 

a 

1 

2 

25 

32 

,00.-, oil 

2 

1 

2a 

27 

-.05-. oi 

9 

9 

-.j.o-,06  • 

10 

10 

-.i5-.il 

2 

2 

-.20-.16 

0 

“.25-. 21 

0 

r.30-,26  .. 

1 

1 

TOTAL 

21 

35 

a5 

20 

289 

aio 
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(2)  The  item-key  correlations  were  cross -validated  on  a new  sample  of  200 
randomly  selected  cases,  but  only  for  items  cn  the  Weapons  and  the 
Engineering  examinations,  and  only  against  key  E100.  Again,  these 
results  are  displayed  in  appropriate  places  in  Table  li»3«  Some  re- 
gression was  apparent,  but  not  enough  to  destroy  the  general  pattern 
of  the  results.  Data  from  the  cross-validation  sample  will  be  discussed 
further,  below. 


Statistical  Parameters  of  Scores  on  the  Factor  Keys 

Table  U.5  shows  the  means,  standard  deviations,  reliabilities,  and  homo- 
geneity coefficients  for  the  various  factor  keys  in  the  sample  used  for  the 
iterative  factor  analysis  (Sample  A)  and  the  cross-validation  sample  (Sample  B), 

Of  particular  concern  were  the  reliabilities  and  homogeneity  coefficients.  The 
various  factor  keys  were  presumed  to  be  fairly  ,,puren  measures  of  certain  aspects 
of  achievement  in  Officer  Candidate  School,  and  they  were  presumed  to  be  at  least 
moderately  reliable — more  reliable  (per  item)  than  the  original  tests.  Further, 
they  were  expected  to  result  in  higher  homogeneity  coefficients  than  the  original 
full  examinations. 

The  data  of  Table  U.5  must  therefore  be  compared  with  those  presented  in 
Table  3.2  of  the  previous  chapter.  The  following  conclusions  may  be  drawn: 

(a)  The  S-keys,  with  the  exception  of  Keys  SUOl  and  Sf>01,  maintained  a higher 
average  level  of  reliability -per -item  than  the  original  examinations.  Part  II  of 
Navigation,  which  had  a per -item  reliability  in  Glass  VII  of  -107.  failed  to  be 
surpassed  by  any  of  the  S-keys,  however.  The  same  statement  may  be  made  with 
respect  to  homogeneity  coefficients,  that  is,  the  b-Keys  in  general  surpassed 
the  original  examinations  but  failed  to  surpass  Part  II  of  Navigation. 
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TABLE  h.5 


MEANS,  STANDARD  DEVIATIONS,  RELIABILITY.  AND  HOMOGENEITY  COEFFICIENTS 
FOR  FACTOR  KEYS  IN  A STANDARDIZATION  SAMPLE  (A)  AND 
A CROSS-VALIDATION  SAMPLE  (B) 

N * 200  IN  EACH  SAMPLE 


Key 

Sample 

No.  of 

Items 

I 

a 

K-R 

(20) 

rel. 

r (1  item) 

sio5 

A 

17 

12.16 

2.01* 

.359 

.032 

.060 

E100  ■ 

A 

21 

Ui.U5 

2.1*2 

.1*60 

.039 

.078 

2100 

B 

21 

lli.6ii 

2.36 

.1*26 

.031* 

.067 

S207 

A . 

13 

8.05 

2.31 

.551 

.086 

.139 

E200 

A 

35 

21*.  72 

1*.68 

.725 

.070 

.111 

E200 

B 

35 

23.86 

l*.2l* 

. .661 

.053 

.085 

S309 

A 

23 

16.3.8 

3.03  . 

.600 

.061 

.110 

E300 

A 

hS 

30.63 

5.59' 

.781 

.073 

.121 

E300 

B 

1*5 

30.08 

1*.71  . 

.61*7  ; 

.039 

.067 

SL01 

A 

15 

8.87 

1.99 

.291 

.027 

.01*7 

EiiOO 

A 

'20 

12.91 

2.80 

.553 

.058 

.102 

EiiOO 

B 

20 

12.70 

2.39 

.353 

.ol*l* 

.027 

s5oi 

A 

11 

7.72 

1.38 

.105 

.011 

.022 

E500 

A 

15 

10.02 

1.77 

.196 

.016 

.037 

(b)  In  Sample  A,  the  per -item  reliabilities  of  the  E-keys  (excluding  E500) 
clearly  surpassed  all  the  per-itea  reliabilities  of  the  original  examinations, 
except  possibly  Part  II  of  Navigation. 

(c)  In  Sample  B,  the  cross-yalidation  sample,  the  per -item -reliabilities  and 
inmogeneity  coefficients  showed  a certain  degree  of  regression  or  "shrinkage, n 

as  one  might  expect  in  a cross-validation  sample.  Nevertheless,  they  remained  at 

i 

levels  which  were  still,  in  general,  higher  than  the  corresponding  statistics  in 
the  complete  tests. 

Hence,  the  factor  keys  seem  to  measure  certain  aspects  of  achievement  in  a 
more  reliable  manner  than  the  original  department  examinations  did.  If  the  factor 
tests  could  be  increased  in  reliability  by  adding  comparable  material,  the  resulting 
tests  would  be  more  satisfactory  measures  of  achievement  than  the  original  depart- 

5 

i 

ment  examinations. 

! ■ 

The  above  statements  must  be  understood  to  apply  to  the  first  four  S-keys 
and  the  corresponding  E-keys.  As  noted  previously,  in  view  of  the  low  reliability 
of  S501  and  E500,  they  were  concluded  to  be  "error  factor"  keys  and  not  worthy  of 
further  investigation. 

Correlational  Data  for  Factor  Keys 

Before  proceeding  to  attempt  an  interpretation  of  what  aspects  of  achievement 
the  factor  keys  may  be  presumed  to  measure,  we  shall  present  all  available 
statistical  data  which  may  aid  in  this  interpretation. 

The  question  of  greatest  interest  is  whether  the  factors  are  truly  independent,  1 

i 

at  least  linearly  independent.  Do  the  factors  measure  different  aspects  of 
achievement?  The  most  direct  way  of  answering  this  question  is  to  examine  the 
intercorrelations  among  the  E-keys,  both  in  the  standardization  sample  and  in  the 
cross-validation  sample.  The  data  are  presented  in  Table  2*. 6 (the  raw  correlation 
coefficients)  and  Table  U.7  (the  correlations  corrected  for  attenuation). 

Very  little  can  be  said  about  the  results  in  Table  i*«6,  for  the  relatively 
low  reliabilities  of  the  E-keys  obscure  the  pattern  of  relationships.  In  general, 
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TABLE  lu6 


CORRELATIONS  AMONG  E-KEIS  IT  THE  FACTOR  ANALYSIS  SAMPLE  (A) 
AND  THE  CROSS -VALIDATION  SAMPLE  (B) 

N - 200  IN  EACH. SAMPLE 

Reliabilities  are  in  the  principal  diagonal  . . 


V SAMPLE 

• - ; -E100 

E200 

E300 

El*00 

aoo 

A 

(.B60) 

.256 

.305 

.;  12 

B 

: C.U26) 

.1*16 

• .371* 

• j-9 

J200 

A 

.256 

(.725) 

.612 

.36ii 

B 

.1*16 

(.661) 

.632 

.U78 

ycxr 

A 

.305 

.612 

(.781) 

.287 

B 

.371* 

.632 

•V 

C.6U7) 

.512 

3J4OO 

A 

.252 

.361* 

.28?  • 

(.553) 

B 

.319 

.1*78 

.512 

(.353) 

TABLE  k.l 

CORRELATIONS  AMONG  E-KETS  (TABLE  1+.6) 
CORRECTED  FOR  ATTENUATION 


:ey 

. SAMPLE, 

E100 

. E200 

E300 

EliOO 

,',100 

A 

1.000 

,14*1* 

.509 

.500 

t . 

B 

1.000 

.783 

.712 

.822 

I 

E200 

A 

. I4I4I* 

1.000 

.812 

.575 

B 

-*0 

CO 

1.000 

.966 

.988 

1300 

A 

.509 

.812 

1.000 

.1*37 

B 

.7n 

.966 

1.000 

1.072 

^1*00 

A 

.500 

.575 

.U37 

1.000 

B 

.821  • 

'.988 

1.072 

1.000 

the  correlations  are  low,  except  between  E200  and  E300,  which  correlate  to  the  ex- 


tent of  ,612  in  Sample  A and  ,632  in  Sample  &•  As  might  have  been  expected,  the 

v,  ■' 

correlations  in  Sample  B,  the  cross-validation  sample,  are  in  every  case  'higher1 
than  in  SamDle  A,  This  is  because  the  iterative  factor-analysis  procedure  cap- 
italized on  error  in  Sample  A,  at  least  to  some  extent,  in  producing  sets  of  items 
which  were  relatively  independent  in  that  sample , In  Sample  B these  errors  have 
been  again  randomized,  with  the  result  that  whatever  inherent  correlations  exist 
among  the  items  are  manifested  in  higher  inter-key  correlations. 

Table  4.7,  containing  correlations  corrected  for  attenuation,  presents  a 
picture  somewhat  unfavorable  to  the  hypothesis  that  the  keys  are  linearly  inde- 
pendent. The  picture  is  not  so  unfavorable  in  Sample  Aj  while  keys  E200  and  E30O 
appear  to  be  highly  similar,  keys  E100  and  EiiOO  are  somewhat  independent  of  each 
other  and  of  the  remaining  keys.  (Of  course,  these  data  must  be  appraised  in  the 
light  of  the  possibility  that  the  estimates  of  reliability  used  in  the  correction 
for  attenuation  are  in  error.)  But  the  picture  is  extremely  unfavorable  to  the 
hypothesis  of  linear  independence  in  Sample  B.  Here,  all  keys  appear  to  have  high 
inherent  relations.  Keys  E200  and  E300  have  a corrected  correlation  of  .966  and 
Ke’r  El|00  correlates  highly  with  all  keys  (its  correlation  of  1.072  witn  Key  EI4.OO 
is  of  course  an  overestimation).  Only  Key  E100  maintains  some  degree  of  in- 
dependence from  the  others.  Again,  however,  the  estimates  of  reliability  may  have 
been  in  error  (i.e.,  toe  low,  thus  resulting  in  excessively  high  corrected  correla- 
tions). This  is  particularly  true  for  Key  400,  the  estimated  reliability  of  which 
dropped  to  .3^3  in  the  cross-validation  sample. 

Our  conclusion  must  be  that  while  there  is  some  evidence  that  the  various 
E-keys  are  linearly  independent  (in  the  sense  which  is  referred  to  in  factor 
analysis),  we  must  admit  that  there  is  a very  strong  general  factor  of  achievement 
running  through  the  E-keys.  Certainly  the  keys  are  not  as  independent  as  was 
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hoped,  Fran  the  standpoint  of  this  evidence,  grave  doubt  is  cast  upon  the  notion 
that  it  is  possible  to  measure  reliably  different  aspects  of  achievement  at  CCS, 

Table  U. 8 presents  extensive  correlational  data  for  Class  VIII  (based  on 
the  iterative-factor  -analysis  sample  of  200  cases),  giving  the  correlations  within 
, and  among  the  fol3.owing  sets  of  variables:  ’■  • 

i (a)  'The  -Bureau  examinations,',  by  ..departments,  (The  correlations  shown  here 
: ; - represent  sampling  variations  from  those  shown  in  one  part  of  Table  3,5). 

(b)  Five  S-keys,  6105,  S207,  S309,  SUOl/  S501. 

(c)  Five  corresponding  E-keys,  E100,'  E200,  E300,  EiiOO,  and  E500, 

' - (d);  Three  nFw  key6,  FI,  F2,  and  F3«  These  were  constructed  from  the  corres- 
ponding E-keya  for  a "special  examination  to  be  given  to  selected  NROTC 
- " ••  " ' classes,  and  will  be  described  and  discussed  in  a later  section  of  this 

>■  report  (Chapter  VI). 

(e)  Five  subtests  of  the  Officer  Candidate  Battery  (OCB):  Verbal  Reasoning, 

Mechanical  Comprehension,  Mathematical  Comprehension,  Relative  Movement, 
i and  Spatial  Relations. 

We  shall  now  discuss  the  findings  which  appear  to  be  of  most  interest  in  these  cor- 
relations, 

(l)  Correlations  Between  Bureau  Examinations  and  S-keys. 

It  must  be  remembered,  first  of  all,  that  in  all  cases  of  correlation  between 
Bureau  examinations  and  special  keys,  we  are  dealing  with  part-whole  relations, 
since  the  special  keys  contained  items  from  the  Bureau  examinations.  Table  ii,9 
shows  the  numbers  of  items  overlapping  between  Bureau  examinations  and  the  S-keys 
and  the  E-keys, 

All  examinations  are  more  highly  correlated  with  Key  S207  and  S309  than  with 


the  other  two.  Thus,  there  is  no  evidence  that  the  keys  differentially  measure  sep- 
arate subject-matters.  Taking  the  keys  separately,  furthermore,  we  find  that  their 
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TABLE  k»9 

NUMBERS  OF  ITEMS  ON  BUREAU  EXAMINATIONS 
OVERLAPPING  WITH  FACTOR  KEYS 


Bureau  Exam. 
(Dec.  «52) 

Total  No. 
of  items 
in  Bureau 
Examination 

sio5 

Frequencies 
S207  S309 

cf  items  on 
Sl;01  E100 

Factor  Keys 
E200  E300 

El*00 

Engineering 

75 

0 

0 

0 

0 

3 

9 

9 

3 

Navigation 

60 

l 

2 

6 

2 

3 

7 

11 

3 

Operations 

75 

5 

h 

n 

« 

5 

5 

4 

9 

k 

Orientation 

75 

7 

2 

5 

3 

6 

3 

7 

5 

Seamanship 

50 

1 

1 

2 

0 

1 

3 

2 

0 

Naval  Weapons 

75 

3 

h 

3 

5 

3 

9 

7 

5 

TOTAL 

U10 

17 

13 

23 

15 

21 

35 

kS 

20 

correlations  with  the  various  examinations  are  approximately  proportional.  The 
correlations  with  the  Operations  examinations  are  most  consistently  the  highest. 
These  results  could  be  explained  on  either  of  two  assumptions.  Either  the 
Bureau  examinations  are  complex  and  contain  different  varieties  of  factor  variance, 
or  the  four  factor  keys  are  really  homogenous.  The  first  of  taese  assumptions  seems 
somewhat  preferable  to  the  second. 

(2)  Correlations  Between  Bureau  Examinations  and  E-keys. 

Since  the  E-keys  are  merely  expansions  of  the  S-keys,  it  would  be  expected 
that  the  pattern  of  correlations  between  Bureau  examinations  and  these  keys  would 


be  similar,  in  most  respects,  to  that  with  the  S-keys.  This  expectation  was  con- 
firmed by  the  data.  Again,  there  was  little  evidence  that  the  keys  differentially 
measured  different  classifications  of  subject-matter. 


(3)  Correlations  Between  Bureau  Examinations  and  OCB  Scores. 

Conaiaeration  of  these  correlations  i a somewhat  outside  the  scope  of  this  chap- 
ter, but  in  view  of  the  interest  which  always  attaches  to  data  concerning  the 
-validity  of  aptitude  teats,  they  will  be- discussed  here.  It  is  felt  that  the  best 
way  of  analysing  these  data  is  in  terms  of  the  multiple  correlations  of  the  OCB 
scores  with  the  several  Bureau  examinations.  Table  U.10  presents  the  pertinent  re- 
sults. 

r 

It  will  be  noted  from  Table  I4.IO  that  the  Engineering  and  the  Weapons  examina- 
tions are  best  predicted  by  the  Mechanical  Comprehension  subtest  of  the  OCB,  with 
relatively  little  being  added  to  the  prediction  by  other  subtests.  These  examina- 
tions admittedly  include  many  items  covering  mechanical  matters.  The  Navigation 
examination  is  best  predicted  by  the  Mathematical  Comprehension  subtest,  with  some 
- contributions  from  Spatial  Relations,  Verbal  Reasoning,  and  Relative  Movement.  The 
Operations,  Orientation,  and  Seamanship  examinations  are  consistently  best  pre- 
dicted by  Verbal  Reasoning}  this  result  accords  with  the  conclusion  one  might  reach 
by  a.., careful  inspection  of  these  examinations,  namely,  that  they  involve,  chiefly, 
verbal  reasoning..  . 

There  are  noticeable  differences  in  the  overall  predictability  of  the  examina- 
tions from  the  OCB.  Engineering,  Operations,  and  Weapons  are  predictable  with 
multiple  correlations  from  .552  to  .670,  while  the  remaining  examinations  have 
multiple  correlations  from  .377  to  .ii03. 

(U)  Correlations  Between  S-keys  and  E-keys. 

These  are  in  general  only  moderate,  as  would  be  desired  for  presumably  inde- 
pendent factor  keys,  except  for  the  high  correlations  between  corresponding  keys 
which  are  due  to  overlaps  of  items. 

Since  the  S-keys  were  merely  stepping  stones  to  the  expanded  E-keys,  there  will 
be  no  further  consideration  of  S-keys.  We  shall  also  omit  mention  of  the  F-keys, 
since  these  were  developed  only  for  a special  purpose  to  be  described  later. 
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($)  Correlations  Between  E-keys  and  0C5  Scores. 

If  the  E-keys  truly  measured  separate  aspects  of  achievement,  it  is  conceivable 
that  they  might  show  different  patterns  of  relatione  with  a series  of  aptitude  tests 
such  as  the  Officer  Candidate  Battery,  The  Officer  Candidate  Battery  consists  of 
five  tests  designed  to  measure  different  aspects  of  ability.  The  correlations  found 
among  them  and  shown  in  Table  li,8  are  only  moderate  and  hence  tend  to  support  the 
notion  that  the  tests  measure  different  traits. 

The  technique  of  multiple  correlation  was  again  used  to  see  whether  the  E-keys 
sorted  out  factors  of  achievement  sufficiently  well  to  be  differentially  predictable 
by  t -he  separate  tests  of  the  Officer  Candidate  Battery,  The  results  are  shown  in 
Table  U.ll. 

It  would  appear  that  the  E-keys  are  not  (in  general)  differentially  predictable 
from  OCB  subtests.  To  ,judge  from  the  beta-weights,  the  first  three  keys  are  most 
weighted  with  Verbal  Reasoning;  at  least.  Verbal  Reasoning  makes  the  greatest  unique 
contribution  to  the  prediction  of  these  keys.  Key  E10Q  is  also  weighted  with  a cer- 
tain amount  of  Spatial  Relations.  Keys  E200  and  E300  are  predicted  to  a substantial 
extent  by  all  the  subtests  of  the  OCB,  each  subtest  making  some  substantial  amount 
of  unique  contribution  to  the  variance  of  Keys  E200  and  E300.  It  could  be  argued 
from  these  results  that  Keys  E200  and  E300  represent  "integrated  knowledge"  because 
they  tap  a number  of  different  areas  of  ability,  but  this  interpretation  is  somewhat 
dangerous  (in  view  of  the  possible  unreliability  of  the  beta-weights)  and  involves  a 
definition  of  "integrated  knowledge"  which  was  not  envisaged  at  the  time  this  project 
was  designed. 

In  general,  the  results  discussed  above  are  fairly  consistent  wioh  the  psychol- 
ogical interpretations  of  the  E-keys  which  will  be  described  below.  Xey  Ei*00,  how- 
ever, presents  a puzzle.  To  judge  from  the  beta-weights  in  Table  It.H,  it  appears  to 
be  weighted  most  heavily  with  Mathematical  Comprehension,  whereas  our  psychological 
interpretation  of  the  factor  would  seem  to  indicate  that  it  measures  simple  memory  for 
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TABLE  I*.  11 

MULTIPLE  CORRELATIONS  AND  BETA  WEIGHTS  FOR  THE  PREDICTION  OF 
E-KEYS  FROM  THE  OFFICER  CANDIDATE  BATTERY 

N ■=  200. in  factor  analysis  sample,  OCS  Class  VIII 


0GB  Sub test 

• X ' 

E100 

• * P 

K E 

E200 

r p 

Y 

E300 

r p 

Elj.00 

P 

Verbal  Reasoning 

.178  ‘ .1U2 

.1*16 

.21*1* 

.397 

.217 

.208 

.085 

■Mechanical  Comprehension 

.137  .01*8 

.1*32 

.181* 

.1*55 

.195 

.256 

.127 

Mathematical  Comprehension 

.109  .011 

.1*51 

.162 

.1*1*5 

.132 

.306 

.202 

Relative  Movement 

.026  -.0 59 

.360 

.133 

.381 

.163 

.176 

.012 

Spatial  Relations 

.177  .131 

.348 

.112 

.393 

.163 

.172 

.017 

Multiple  R 

.231* 

.585 

.600 

.31*2 

isolated  facts.  Nevertheless,  the  beta-weights  on  which  the  former  interpretation 
is  ba3ed  are  subject  to  sampling  errors. 

Psychological  Interpretation  of  the  E-keys 

It  has  been  seen,  up  to  this  point,  that  the  evidence  for  differences  among 
the  achievement  factors  measured  by  the  E~keys  is  rather  slim.  Furthermore,  the 
E-keys  do  not  show  any  important  differences  in  their  patterns  of  correlations 
with  the  Bureau  examinations  or  with  Officer  Candidate  Battery  sub-tests.  It  is 
therefore  only  with  extreme  hesitation  that  an  effort  is  made  here  tc  offer 
psychological  interpretations  of  the  E-keys. 

Two  members  of  .the  project  staff  examined  the  content  of  the  E-keys  with  a 
view  towards  psychological  interpretation.  The  approaches  taken  by  these  two 
persons  were  quite  different,  but  led  to  somewhat  similar  conclusions, 

• Staff  member  A examined  the  da4- a simply  With  the  purpose  of  judging  what 
mental  processes  the  items  of  a given  key  required  in  common.  His  approach  was 


thus  similar  to  that  which  is  ordinarily  taken  in  interpreting  any  results  obtained 
by  factor  analysis.  The  interpretation  made  by  this  staff  member  is  as  follows: 

Key  E100.  This  key  appears  to  represent  a type  of  verbal  reasoning  in 

which  judgments  are  made.  The  examinee  is  presented  with  agproblem  about 
which  he  makes  a judgment,  utilizing  any  information  he  may  have,  and  at 
the  same  time  using  a variety  of  "common  sense"  in  "figuring  out"  what  the 
intended  answer  is  on  the  basis  of  the  structure  of  the  alternatives  pre- 
sented. Very  few  of  the  items  seem  to  require  spatial,  mechanical,  or 
mathematical  reasoning.  Most  of  them  do,  however,  involve  factual  inform- 
ation about  the  Navy  and  related  subjects. 

Key  E200.  This  key  appears  to  involve  spatial  and  temporal  reasoning,  that 
is,  reasoning  in  problems  involving  space,  time,  and  sequences  of  events. 
Problems  requiring  mechanical  reasoning  are  included  in  this  definition. 

Key  E300.  This  key  appears  highly  similar  to  Key  E200,  but  tends  to  em- 
phasize mathematical  and  numerical  ability  more  than  Key  E200.  It  would 
be  expected  to  correlate  highly  with  Key  E200.  (This  statement  was  made 
without  knowledge  of  the  actual  correlation.) 

Key  EljOQ.  This  key  appears  to  involve  nothing  more  than  the  memory  for  iso- 
lated bits  of  information,  that  is,  factual  information.  If  anything,  one 
would  expect  the  scholarly,  bookish  type  of  individual  who  likes  to  re- 

■h 

member  trivial  details  to  get  high  scores  on  this  key. 

Staff  member  B decided  to  classify  the  items  of  the  various  keys  with  respect 
to  the  OCB  subtests  with  which  he  thought  each  item  would  correlate  most  highly. 
This  was  done  solely  on  the  basis  of  the  names  of  these  subtests,  since  the  actual 
test  forms  of  the  OCB  were  no*’  available  at  the  time  this  classification  was  done. 
Table  4.12  shows  the  classification  made  by  this  staff  member. 

It  is  apparent  that  the  two  staff  members  agreed  that  Keys  E200  and  E300  in- 
cluded larger  numbers  of  mathematical  and  mechanical  Items  than  the  remaining  two 


TABLE  ii.i2 

SUBJECTIVE  CLASSIFICATION  A STAFF  MEMBER  OF  ITEMS 


OF  THE  E-KEYS  WITH  RESPECT 

TO  COB 

SUBTESTS 

. , . 

FREQUENCY 

OCB  Sub test 

ElOO 

E20Q 

E300 

EiiOQ 

Verbal  Reasoning 

12 

iu 

16 

11 

Mathematical  Comprehension 

5 

10 

11 

2 

Mechanical  Comprehension 

2 

10 

12 

7 

Spatial  Relations  and  Relative  Movement  ■ 

. 2 

1 

6 

0 

TOTAL 

21 

35 

U5 

20 

keys.  They  also  agreed  that  Key  ElOO  measured  verbal  reasoning  more  than  anything 
else.  Beyond  this,  staff  member  B was  handicapped  in'  using  the  categories  of  the 
OCB.  Using  these  categories,  he  could  not  have  reached  the  conclusion  that  El*00 
measured  simple  memory  for  factual  information;  instead,  he  was  forced  to  conclude 
that  it  chiefly  measured  verbal  reasoning.  On  the  whole,  the  two  staff  members 
agreed  in  their  interpretations ' of  the  E keys,  a fact  which  lends  some  support  to 
the  notion  that  the  E-keys  are  measuring  different  aspects  of  achievement  in  the 
officer  candidate  curriculum. 

Concluding  Statement 

The  research  reported  in  this  chapter  attempted  to  sort  out  the  items  of 
the  Bureau  OCS  examinations  into  a number  of  sets  measuring  different  aspects 
achievement.  The  research  was  only  partially  successful  in  attaining  this  ob- 
jective. The  sets  of  items,  while  appearing  to  be  different  from  a subjective 
point  of  view,  were  not  sufficiently  independent  from  a statistical  point  of  view. 
The  question  may  now  be  raised;  what  do  the  results  show  with  regard  to 
"integrated  knowledge"?  It  may  be  suggested  that  no  one  of  the  E~keys  measures 

- 87  - 


! anything  distinctly  recognizable  as  "integrated  knowledge" ; instead,  all  of  them 

fl 

do,  at  least  keys  E100,  E200,  and  E300.  Only  key  EiiOO  appears  (at  least  to  one 
ki 

y staff  member)  to  measure  simple  factual  information.  Keys  E200  and  E300  (which 

• ar8  in  any  case  highly  correlated)  appear  to  measure  the  technical  and  mathematical 

| kinds  of  subject  matter  which  are  so  predominant  in  the  officer  candidate  curricu- 

I 

lum.  Key  E100,  heaver,  appears  to  measure  something  slightly  different,  namely, 
judgmental  problem-solving.  This  aspect  of  achievement  is  undoubtedly  of  interest 

i 

to  the  Navy. 

* i 

| It  is  recommended  that  the  E-keys  would  be  of  value  in  future  research  in  at 

'I  least  two  respects: 

* 

>|  (1)  as  sets  of  items  against  which  n6w  items  could  be  validated;  and 

(2)  as  predictors  of  later  success  in  the  Navy. 

It  is  conceivable,  for  example,  that  key  E100  measures  those  aspects  of  problem- 
solving which  are  most  required  in  the  novel  situations  which  the  naval  officer 

I 

is  constantly  encountering. 

For  either  of  these  purposes,  it  might  be  well  to  expand  the  E-keys  even 
further.  That  is,  items  with  validities  against  the  E-keys  somewhat  lower  than 

i 

.30  (say,  down  to  .20)  could  be  added  to  the  keys.  Reference  may  be  had  to  Table 
U.3  in  selecting  such  items.  Items  should,  however,  be  added  to  the  E-keys  in 
such  a way  that  there  is  no  overlap  among  the  newly-expanded  keys. 


1 


"7 


■w 


THE  TECHNIQUE  OF  OHBRAZIRO  OTIMCTHOSTICS 


Introduction 


As  as  introduction  to  this  chapter,  it  may  he  pertinent  to  quote  from  a 


chapter  found  in  a recent  textbook  cox  educational 


"Most  educational  acMcrenrrt  tests  sore  concenxeA  primarily  with  power  or 
level  rather  than,  rate.  There  is  sase  theoretical  basis  for  co; iterating;  ^bat  the 
performance  an  a power  test  should  he  exj.  eased  not  in  terms  of  raw  scores,  hat  in 
terns  of  the  Irrel  of  difficulty  of  the  u st  difficult  group  of  liens  to  which  the 
esssiage  is  able  to  respond  correctly  some  set  pocqpctrtloai  of  Idle  tine,  such  as  $0 
percent.  The  test  night  consist;  for  •example,  of  ten  sets  of  items  grouped  accord 
tag  to  difficulty,  each  successive  set  being  mere  difficult  than  the  set  preceding 
An  individual^  scare  cat  this  tent  nay  be  J,2f  meaning  that  he  is  able  to  respond 
correctly  to  approximately  half  of  the  items  In  the  set  that  has  been  given  a 
scale  value  of  J,2,  but  tint  he  responds  correctly  to  fewer  than,  half  of  the  items 
in  the  higher  sets  and  to  none  than  half  in  the  loser.  There  are  a number  of  very 
serious  practical  sad  theoretical  difficulties  in  constructing  such  tests,  one  of 
the  most  troublesome  being  that  the  items  must  be  highly  homogeneous,  or  highly 
correlated  with  one  another.  Variations  in  the  learning  experiences  of  different 
pupils  are  such  that  few  types  of  items  can  be  expected  to  be  hanogeneouo,  or  to 
jaii.nta.in  a stable  comparative  difficulty.  Tor  this  and  other  reasons,  tests  re- 
porting performance  in  terms  of  level  of  difficulty  are  rarely  constructed  in  edu- 


cational measurement. 


Despite  the  difficulties  pointed  out  in  this  quotation,  it  is  believed  that 


a technique  of  scaling  scarce  in  terms  of  item  difficulties  has  ucrtr  teen  given  the 


attention  it  deserves.  It  Is  also  believed  that  the  results  -to  be  achieved  can 


have  important  practical  iaplicatians  for -the  interpretation  and  use  of  tests. 


In  the  case  of  the  present  research,  use  of  this  technique  can  rafee  results  asare 


meaningful  to  the  Hscvy.  This  chapter,  therefore,  will  sketch  the  theoretical 


basis  for  the  technique  and  will  report  and  Interpret  ease  results  attained  by 


Its  use. 


* Lindquist,  E.  F.  (editor)  TMuratlanal.  Measurement.  Washington,  D.C.: 
American  Council  on  Education,  19^1.  ikls  'quotation  is  from  page  70 6. 
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IhogretlcaO.  Baals 

It  lit  f truly  awraaed  thaf.  tmtta  umaarttired  by  psychcflogliafcl  Mad  efasatimaal  torts 
eadst  alxnc  castinu*,  that  is,  tint  the  traits  eadst  in  various  aacnarts  izr  different 
individuals.  Ter  any  given.  trait,  therefore,  ire  sqr  represent  the  amount  of  that  trait* 
along  the  Tsaaellae  of  a graph, 

Ih  the  ease  of  teste  af  ability  vfaicdr  trmxLve  the  "passing*  or  "failing*  of  items, 
no  nay  also  he  concerned  rlth  the  relation  between  the  aaaogfc  of  the  trait  utrLch  an 

individual  possesses  and  the  jrmhaht.lt  I ji  of  Ms  passing  itaas  at  marioca  levels  of  dif  ~ 

♦ . • , • * 

•/*  ficulty,  We  say  assume  that  individuals  -with  high  ability  Trill  pass  items  of  high  dif- 
ficulty sith  e higher  probability  than  an.  individual  of  lor  ability. 

If  se  further  assume  that  all  items  l2t  a given  set  measure  the  same  trait,  se  may 
arrange  these  item*  along  the  scale  of  ability  mentioned  earlier,  so  that  the  items 
themselves  define  -various  amounts  of  ability.  Thus,  Mgh-^fficulty  items  define  the 
upper  reaches  of  the  scale  of  ability  in  the  sense  that  higfc-abiUty  ind.1rldna.la  are 
Shis  to  pass  them  sith  sense  specified  probability  (e.g,  ,5)»  likewise,  lcsr-difficulty 
Items  define  the  loser  reaches  of  the  scale  of  ability  in  the  sense  that  even  individ- 
uals of  lor  ability  are  able  ter  pass  them  with  the  specified  probability. 

Ia  practice,  the  best  estimate  se  can  make  of  an  individual**  ability  is  in  terms 
of  his  score  (e.g.,  number  right)  an*  set  of  Items.  Brin  score  has  the  fraction  of 
suaaating  the  probabilities  with  which  he  can.  pass  the  'various  items  in.  a set.  In 
fact,  the  score  Is  a direct  function  of  the  average  probability  with  which  he  can  pass 
item*  in  the  set.  If,  however,  se  determine  or  estimate  in  some  way  thee  probabilities 
slth  which  an  individual  slth  a gives  score  can  pass  items  of  "various  difficulties,  se 
can  make  the  meaning  of  this  score  more  apparent.  Graphically,  se  may  draw  a curve 
representing  these  probabilities.  Such  a curve  is  called,  an  individual  operating 
■^•rscterlgtic  (mot  to  he  confused  with  the  ifacp  typerating  characteristics  studied  by 
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lord.)*  Figure  5*1  Illustrates  ar  idealised  Individual  operating  cbarmc teristic . 
Ihe  curve  is  to  be  interpreted  as  shoeing  that  an  individual  with  a certain  score 
on.  a test  (say#  £©)  'will  pass  items  of  -f&ji  difficulty  (for  the  group)  with  a 
probability  of  .965j  items  of  difficulty,  with  a probability  of  .25;  item  of 
10$  difficulty,  with  & probability  of  .880;  etc.  For  the  purpOBeB  of  graphical 
presentation,  some  shape  for  the  operating  characteristic  curre  has  to  be  assumed. 
There  is  good  reaaen  to  assume,  in  practice,  that  operating  characteristic  curves 
have  the  shape  af  Trorrml  ogtvwr. 

It  is,  however,  possible  to  conceive  a type  of  test,  or  a type  of  trait, 
that  would,  hare  what  we  shall  call  a perfect  operating  characteristic . A perfect 
individual  operating  characteristic  la  illustrated  in  Figure  5.2,  and  nay  be  in- 
terpreted by  stating  that  an  individual  with  a certain  degree  of  ability  will 
pass  all  items  up  to  a certain  level  of  difficulty,  but  will  fail  all  items  be- 
yond that  level  of  difficulty.  If  all  individual  operating  characteristics  are 
of  this  shape,  the  test  would  be  precisely-  what  Loeringer  demands  far  a "per- 
fectly homogeneous  test."  In  fact,  a perfectly  homogeneous  test  absolutely  implies 
that  all  operating  characteristics  be  perfect.**  It  is  also  true  of  this  type  of 
test  that  all  items  would  have  tetracharic  intercorrelatiooas  of  unity. 

It  is,  however,  inconceivable  that  tests  or  traits  having  perfect  operating 
characteristics  would  ever  be  found  in  practice.  Even  in  the  most  carefully  con- 
trolled psychophysical  experiments,  the  psychophysical  curves  (which  are  analogous 
to  our  individual  operating  characteristic  curves)  do  not  have  the  shape  shown  in 

* Lord,  F.  M.  A theory  of  test  scores.  Psychometric  Monographs,  lc.  7» 

**  Or  nearly  bo.  The  requirement  is  actually  that  the  Indivicnial  operating 
characteristic  curves  always  move  from  unity  at  a given  level  of  difficulty 
to  zero  for  the  item  of  the  next  higher  level  of  difficulty.  Thus,  the  central 
portion  or  the  operating  characteristic  curves  could  depart  slightly  from  a 
perfectly  vertical  position  when  two  adjacent  items  are  slightly  different  in 
difficulty. 
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Probability  of  Passing  Probability 


*/■* 

< .7 


.5 

A 

.3 


.2 


-2-10123 
Item  Difficulty  (Normal  Deviate  Form ) 

Figure  5.1.  Operating  Characteristic  Curve  For  an  Individual 
with  a Score  of  20.  (Hypothetical  Data) 

The  curve  shows  the  probability  which  the  individual 
has  of  passing  items  of  any  given  degree  of  difficulty. 

S 

] 

* 

• 9 


-5  -2-1  0 1 2 3 


Item  Difficulty  ( Norm*!  De  visit  Form) 


Figure  5«2.  A perfect  operating  characteristic  curve. 
The  individual ' s score  is  still  20. 


* 
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figure  5.i 2,  Furthermore,  perfect  operating  characteristic  curves  would  not  be  found 
in  the  case  of  mechanical  or  electronic  equipment  (e.g.,  a vacuum  tube  in  a relay 
circuit).  It  therefore  seems  reasonable  to  conclude  that  the  perfect  operating 
characteristic  represents  an  impossible  ideal,  and  that  meaningful  results  can  be 
attained  with  tests  having  imperfect  operating  characteristic  curves.  The  quota- 
tion from  Lindquist  cited  above  seems  to  impi^  that  the  scaling  of  scores  on 
achievement  tests  with  respect  to  item  difficulties  cannot  be  done  unless  the  op- 
erating characteristics  are  perfect  or  nearly  perfectj  the  present  proposal  assumes, 
however,  that  meaningful  scaling  can  be  done  even  when  the  operating  character- 
istics are  imperfect,  that  is,  when  the  slopes  are  not  vertical  but  assume  the 
shape  of  something  like  a normal  ogive. 

The  technique  we  shall  employ  will  involve  the  estimation  of  probabilities-of  * 
passing  for  various  test  scores  and  for  items  of  various  difficulties.  Instead  of 
using  individual  operating  characteristic  curves,  however,  we  shall  find  average 
individual  operating  characteristic  (O.C.)  curves  for  all  individuals  making  a 
specified  observed  score  on  the  test.  Furthermore,  we  shall  find  the  probabilities 
of  passing  for  groups  of  items  of  similar  difficulties,  in  view  of  the  fact  that  a 
very  high  N would  be  required  to  estimate  the  probabilities  for  individual  items 
with  precision.  The  average  0.  C.  curves  will  be  plotted  on  a probability  graph, 
the  item  difficulties  being  scaled  on  the  abscissa  in  terms  of  their  normal  deviate 
values,  and  the  probabilities  of  passing  being  plotted  on  the  ordinate,  likewise  in 


v axuco« 


In  this  wajr , the  plot  Vu-il  t tiveal  oxie  ex- 


tent to  which  the  0.  C.  curves  conform  to  the  assumption  that  they  have  the  shape 
of  normal  ogives,  and  further,  the  extent  to  which  they  have  identical  slopes.  If 
all  averaged  individual  0.  C.  curves  have  the  same  normal  ogive  shape  and  slope, 
the  resulting  plot  will  look  like  Figure  5.3. 

Same  question  might  be  raised  about  the  use  of  the  observed  score  as  the  basis 
for  an  0.  C.  curve.  It  would,  of  course,  be  better  to  use  true  scores.  However,  the 
true  scores  are  obviously  inaccessible,  and  even  "regressed"  scores  (as  estimates  of 
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/few  Difficulty  Deviate  Farrnj 

Figure  5*3 • Operating  Characteristic  Curves  for 

various  score  levels  on  a hypothetical. 

50  item  test. 


J 


> 


the  true  score)  are  simply  linear  functions  of  the  observed  scores.  Hence,  the 
observed  scores  can  be  used  with  justification. 

The  slope  of  the  0*  C,  curve  when  plotted  on  probability  paper  (i.e.  with 
both  coordinates  in  normal  deviate  form)  should  presumably  have  something  to  do 
with  the  homogeneity  of  a test.  This  matter  has  not  been  thoroughly  Investigated, 
due  to  lack  of  time,  either  from  an  empirical  or  a mathematical  standpoint.  It  is 
obvious,  however,  that  the  0.  C.  curves  for  a perfectly  homogeneous  test  would 
appear  as  perfectly  vertical  lines  of  the  0.  C.  plot  on  probability  paper.  Fur- 
ther, for  a perfectly  heterogeneous  test,  the  variance  of  scores  would  be  solely 
error  variance,  and  insofar  as  the  items  might  vary  in  difficulty  the  0.  C.  curves 
would  parallel  the  curve  for  the  aver.uge  individual  in  the  total  group.  Thus,  the 
nrinlww.1  operating  characteristic  would  be  represented  on  the  G.  C„  plot  with  a 45° 
slope  such  that  the  normal  deviate  value  of  the  baseline  would  equal  the  norma], 
deviate  value  of  the  abscissa.  An  0.  C.  with  this  minimal  slope  has  been  plotted 
on  Figure  5*3«  The  homogeneity  of  a test  is  therefore  a function  of  the  extent  to 
which  its  0.  C.  curves  tend  to  tilt  toward  tne  vertical,  measuring  from  the  mini- 
mal 4-5°  position.  As  a tentative  measure  of  homogeneity  or  operatijjg  character- 
istic, one  might  suggest  the  measure 

* - 45° 

45° 

where  c*  is  the  average  angle  of  the  slope  of  the  0.  C.  curves.  For  the  investi- 
gation of  the  slopes  of  the  0.  C.  curves  which  will  be  presented  below,  it  would 
be  necessary  to  apply  or  devise  some  sort  of  curve-fitting  technique,  e.g.  some 
technique  akin  to  those  used  in  psychophysics,  in  order  to  find  the  average  angle 
of  slope,  a<  . This  procedure  would  undoubtedly  be  extremely  laborious,  however. 
The  estimates  or  reliability  and  homogeneity  male  elsewhere  in  this  report  are 
regarded  as  sufficiently  useful  to  make  unnecessary  the  laborious  investigation 
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of  0.  C.  slopes  which  might  otherwise  he  made.  It  will  be  apparent,  however,  that 
the  operating  characteristics  of  most  of  the  curves  which  will  be  shown  here  are  ex- 
tremely lor.  This  fact  dees  not  vitiate  the  utility  of  the  technique  of  operating 
characteristics . 

Importance  for  the  Mary 

The  importance  of  this  technique  for  the  Havy  may  nor  be  stated.  The  scores  tan 
nearly  all  educational  and  psychological  tests  are  usually  Interpreted  in  terms  of 
relative,  group  standards*  Far  example,  a score  on  a certain  test  might  he  said  to 
have  a percentile  rank  of  65,  or  a Navy  standard  score  of  54,  or  a rtanine  of  6,  etc., 
hut  always  with  respect  to  the  actual  ability  of  the  examinee:  it  does  not  specify 
the  kinds  of  Items  the  examinee  can  pass,  nor  the  probabilities  with  which  he  will 
pass  them. 

Suppose,  therefore,  one  is  Interested  in  establishing  a critical  score  such  that 
students  below  that  score  will  not  he  considered  to  have  passed  a naval  officer  candi- 
date course.  If  one  uses  only  relative  standards,  the  decision  becomes  purely  arbi- 
trary. With,  the  vise  of  0.  C.  curves,  however,  one  can  find  that  scare  which  seems  to 
stand  at  the  critical  point  between  acceptable  and  unacceptable  candidates,  in  terms 
of  what  these  candidates  know  or  can  do.  For  this  purpose,  the  0.  C.  graphs  must  be 
used.  In  conjunction  with  the  actual  items  on  the  test,  preferably  ranked  in  difficulty 
to  correspond  to  the  scaling  of  the  items  an  the  baseline  of  the  0.  C.  plot. 

It  is  suggested  that  the  items  be  inspected  with  a view  to  finding  the  point  on 
the  item  difficulty  scale  where  it  is  desired  that  the  minimally  acceptable  candidate 
will  be  able  to  pass  the  items  with  same  specified  probability,  say  .75  (that  is,  3 
times  out  of  4,  or  75^  of  the  time).  The  0.  C.  graph  can  then  be  used  to  find  the 
test  scare  which  corresponds  to  this  requirement.  Far  example,  suppose  after  Inspec- 
tion of  the  actual  items  represented  along  the  baseline  of  Figure  5*3 > it  is  decided. 
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that  minimally  acceptable  candidate*  should  be  able  to  pass,  75&  of  the  time, 
act  item  with  a scale  Talus  of  ••6k.  It  Till  he  seen  from  Figure  5.3  that  this 
requirement  correspcnds  most  closely  to  a score  of  21.  By  reference  to  the  dis- 
tribution of  scores,  the  percentage  attaining  this  critical  score  in  any  speci- 
fied group  can,  of  course,  be  determined,  but  it  is  believed  mare  meaningful  to 
first  detendne  the  critical  sccre  by  reference  tc  the  items  rather  than  by  refer- 
ence to  the  percentage  attaining  a given  critical  score. 

The  utility  of  the  technique  of  0,  C.  curves  in  compering  groups  is  also 
apparent.  Once  a critical  sccre  is  established  (without  reference  to  the  distri- 
bution of  scores)  the  percentages  attaining  the  critical  score  In  various  groups 
can  be  determined  and  compared.  This  procedure  wl.ll  be  illustrated  in  Chapter  6 
of  this  report,  where  OCS  and  !G?.C7C  groups  are  compared. 


Operating  Characteristic  Curves  for  E-beys  in  003  Class  VIII 

The  technique  of  operating  characteristics  is  probably  not  appropriate  unless 
the  items  analyzed  have  at  least  sane  degree  of  homogeneity.  It  is  believed  that 
the  iterative  factor-analysis  techniques  employed  in  the  analysis  of  the  OCS  ex- 
aminations produced  sets  of  items  (the  ’'E-keys” ) sufficiently  homogeneous  tc 
Justify  the  use  of  0,  C.  analysis.  It  will  be  recalled  that  each  item  In  the 
E-keys  had  a detractor ic  correlation  of  .30  or  greater  with  its  respective  E-key. 
This  Insures  that  item  0.  C.'s  have  at  least  moderate  slopes  and  hence  in  general 
insures  the  separation  of  individual  0.  C.!s.* 


* in  item  0.  C.  represents  the  relation  betifeen  ability  (as  estimated  by  the 
observed  score ) and  the  proportions  passing  a given,  item.  Thus,  the  item  0.  C. 
curve  is  represented  by  the  intercepts  of  individual  0.  C.  fs  with  the  perpendicu- 
lar at  a given  point  on  the  baseline  of  our  6.  C.  plots. 
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The  0.  C.  plots  for  keys  KLOQ,  E200,  S3Q0,  and  fit  00  «s  applied  to  OCS  Class  TIH 
exsidnatians  are  shown  in  Figures  5*4,  5.5,  3,6,  and  5*7  respectively*  Tables  5*1  to 
5*4-  five  the  data  m.  which  the  curves  are  based  and  identify  the  items  scaled  along 
the  base-line.  The  total  number  of  cases  for  each  plot  is  200;  the  iterative  factor- 
analysis  sample  vu  used. 

Since  the  use  of  these  plots  involves  reference  to  the  actual  item,  which  are 
under  a security  classification,  it  is  not  possible  to  discuss  the  implications  of 
these  plots  beyond  pointing  out  that,  the  0.  C.  curves  conform  reasonably  veil  to  the 
hypothesis  that  they  are  normal  ogives  (i.e.  straight  liras  when,  plotted  an  corral 
deviate  coordinates).  The  parallelism  among  the  0.  C.  curves  for  the  various  scares 
Should  be  pointed  out. 

It  may  be  concluded  that  a test  scare  is  shown  to  have  high  validity  in  predict- 
ing the  probability  with  which  sen  individual  will  pass  an  item  of  a given  difficulty, 
that  the  relationship  between  item  difficulty  and  probability  of  passing  can  be  ex- 
pressed as  a .linear  function,  and  that  the  0.  C.  curves  form  a family  of  curves  whose 
separation  can  be  described  in  terms  of  a single  paraneter. 

The  occasional,  instances  where  the  0.  C.  curves  cross  occur  chiefly  at  the  ex-  . 
tremea  of  difficulty,  where  the  estimates  of  probability  of  passing  are  undoubtedly 
less  reliable.  An  efibrt  was  made  to  make  them  as  reliable  as  possible  by  basing 
them  an  reasonably  large  numbers  of  cases.  This  was  accomplished  by  grouping  scores 
toward  the  ends  of  the  distribution. 

As  was  pointed  out  earlier,  the  operating  characteristics  found  for  these  keyB  are 
extremely  low,  in  the  sense  that  their  elopes  are  almost  insensibly  greater  than  the 
45°  slope  which  would  obtain  far  a perfectly  heterogeneous  test.  It  will  be  remembered 
that  the  homogeneity  coefficients  computed  for  these  keys  and  reported  in  Chapter  4 
are  uniformly  low.  This  means  that  the  tetracharic  correlations  among  the  items  are 


Item  Difficulty  (Kcrml  Deviate  Form) 


Figure  5,4.  Jbreraged  irkLiridual  0.  C.'s  far  -rarioas  score  groups 
ota  Key  E1Q0.  The  'broken  Use  extending  diagonally 
across  the  abode  plot  represents  the  expected  minimal 
slope  of  the  0.  C,  carre  for  a perfectly  heterogeneous 
teats  The  data  fear  this  figure  are  gtrss  is  Table  5»1 
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Figure  5.5. 


Item  Difficulty  (Normal  Deviate  Form) 

Averaged  individual  0.  C.  's  for  various  score  groups 
on  Key  E200.  The  broken  line  extending  diagonally 
across  the  vbole  plot  represents  the  expected  adnlaal 
slope  of  the  0.  C.  curve  for  a perfectly  heterogeneous 
test.  ’The  data  far  this  figure  are  given  in  Table  5*2 


; • ST  f *■  » A r I 

-/.o  -l.u  -.3  u .3  I.U 

Item  Difficulty  (Norm!  Deviate  Form) 

Figure  5.6.  Averaged  iTidiTidual  0.  C.  Te  far  'various  score  greets 
on  Key  E300.  The  broken  Line  extending  diagonally 
across  the  -whole  pint  represents  the  expected,  mtnijaal. 
slope  of  the  0.  C.  curve  focr  a perfectly  heterogeneous 
test.  The  data  far  this  figure  are  given  in.  Table  5»3» 


< 
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Figure  5.7.  Averaged  individual  C.C.'s  for  various  score  groups 
on  Key  E400.  lhe  broken  line  extending  diagonally 
across  the  whole  pint  represents  the  expected  mi  rriwtl 
slope  of  the  0.  C.  curve  for  a perfectly  heterogeneous 
test.  ‘The  data  for  this  figure  are  given  In  Table  5.4, 
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TABLE  5.1 


DATA  ?0R  OPHIATUG  C2ARACTKRISTIC  CURTE8,  XIX  XLOO, 

oca  class  mi  ixakhutiors 


TABLE  >.2 


DATA  TOR  0PKRA3TK  CHARACTHUSTIC  CURVES,  KEY  E2C0, 
OCS  CLASS  YIH  EXAJCCKATIQNB 

Total  8 * 200  cases  in  I ter  at  ire  fact  or -analysis  sample 


Item 

Group 

12-17 

15.6 

Scare  Groups  (Rshge  and  Mean)  (Rights ) 

18-20  21-22  23-24  25-26  ■'  27-28 

19.1  • 21.4  23.3  25.4.  27.5 

29-34 

30.2 

A 

• .73 

.046 

.183 

■ .185 

.OlB 

.267 

.333 

.514 

B 

.24 

.250 

.258 

.296 

.357 

.375 

•5t4 

.646 

C 

-.24 

.325 

.410 

.471 

,561 

.671 

.751 

.849 

D 

-,63 

.428 

.608 

.676 

.732 

.828 

.840 

.917 

E 

-*.95 

.700 

.660 

.800 

.900 

.833 

•9U- 

.950 

1 

-l.lfi 

.693 

.825 

.852 

.920 

.900 

.917 

.986 

Ho.  of  cases  in 

' 

score 

group: 

22 

30 

27 

28 

30 

27 

36 

SPECIFICATION  OF  HEM  CROUPS 


(The  itm  numbers  on  the  respective  examinations  and  the  total  proportion  passing 
are  given) 


item:  group 


ABC  D E F* 


P # P 

# P 

# 

p 

p 

# P 

H-54  .180  0*-9  .370 

E-53  .515 

1-15 

.690 

w-50 

.820 

Op-13  .850 

Op-40  .295  H-33  .410 

W-47  .565 

op-25 

.700 

H-20 

.825 

N*5  .865 

0-46  .415 

S-30  .565 

H-25 

.710 

W-59 

.830 

s-18  .890 

0-58  .430 

E-36  . 585 

E-13 

.715 

Op -20 

.835 

H-17  .910 

b4o  .600 

E-lB 

.725 

E-73 

.835 

E-37  .660 

v-k9 

.735 

H m TttarUBKtim 

JWj6  .670 

E-50 

.740 

W w Earn  Rfespona 

y-c4 

.745 

*V-20, 

vith  a p of 

Op  m Operatiiots 

>4*52 

«755 

.970  **e  omitted 

0 * Orientation 

v-69 

•765 

from  this  group. 

E k Engineering 

x-15 

,7^ 

S * Seamanship 

W$i 

•7T® 

\ I 

!. 

j- 

( 


■ , ' 


A • 

.66 

.0U5 

.136 

.153 

.218 

.248 

.307 

•391 

.600 

B 

.13 

.164 

.300 

.258 

.290 

.469 

.584 

.627 

.770 

C 

-.04 

.247 

.319 

.452 

.458 

<563 

.651 

.665 

.800 

V 

>-.30 

.356 

.439 

.576 

.615 

.590 

.693 

.819 

.908 

E 

-.71 

.545 

.608 

.677 

.746 

.832 

.830 

.807 

.944 

F 

-1.07 

.692 

.707 

• 

00 

0 

H 

.854 

.914 

.942 

.957 

,944 

G 

-1.4l 

.764 

.882 

.883 

.938 

.966 

.976 

.965 

.970 

No.  of 

cases  in 

CAAWO 
MWSM  *—  , 

group: 

22 

22 

24 

29 

35 

25 

23 

20 

(The  item  numbers  on  the  respective 
are  given) 


SPECIFICATION  OF  ITEM  GROUPS 

spective  examinations  and  the  total  proportion  passing 


ITEM  GROUP 
D 

# P 


F 

# P 


G* 

# P 


Op-26 

.235 

W-31 

.405 

e-4 

.505 

op-17 

.585 

E-31 

.705 

N-52 

0825 

v-7 

.905 

N-60 

.240 

N-59 

.415 

Qp-30 

.510 

E-3 

.585 

Op-ll 

.715 

0-70 

.830 

E-4l 

.915 

v C 

r\or\ 

n^  m 

li  on 

• TC.V 

TJ  £)■ 
n-wv 

ct  e; 

nr_oo 

hoc; 

“ S.s  S 

TJ— 37 
*’ 

■ I w*/ 

0-l6 

.8^s 
* ~ 

E-68 

.Ql'i 

- ■>*  — *’  1 

Op-6 

.420 

E-25 

.515 

S-13 

.<635 

E-59 

.755 

0-4 

.850 

0p’->2a- 

.930  | 

N-58 

.485 

s-4 

.525 

0-72 

.655 

Jl-4 

.784 

E-58 

.860 

N-42 

.935 

N-7 

.530 

N-56 

.660 

Op -8 

.785 

v-27 

.865 

N-32 

.530 

0p~12 

•795 

R-3 

.880 

1 | 

W-43 

.800 

0-28 

.880 

» 

i 1 

* 0-56  and  W-10,  with  p's  of 
group. 


0-15  .385 

->n  ftnfS  - ?7^ j respecHvely"  ^ere  oaittel  frcmtnig 
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m 5.4 


mei  pgr  cestmoD  csAJutemime  curtej,  nr  tim, 

oc8  class  mi  siiocxmais 

Total  1 ps  2CO  cases  ; ' iterative  faxrtar*aa»ljr8is  sample 


T-rser* 

Proportion  Passing 

1UCA 

Group 

normal 

Scare  Groups  (Range  and  Mean)  (Rights) 

deviate 

6*9 

10-11 

12 

13 

14 

15 

16 

17-20 

value 

8,4 

10.7 

12.0 

.33.0 

l4.'w 

15.0 

16,0 

1JA 

A 

,te 

.232 

.179 

.214 

.315 

.360 

.46? 

.443 

•636 

B 

-.03 

.241 

.350 

.473 

.478 

.550 

.569 

.761 

C 

..4l 

•345 

.51^ 

.655 

.652 

.773 

.745 

.894 

.079 

B 

-.68 

.1)20 

.679 

.741 

.783 

.830 

.997 

.898 

.943 

E 

-1,32 

•757 

.857 

.864 

.9 48 

.952 

.976 

.902 

.991 

Ho,  of  cases  la 
scare  group:  28 


35 


28 


23 


25 


17 


22 


22 


SPHUTICAjnaH  OF  HEN  CROUPS 

(Toe  item  numbers  an  the  respective  examination  and  the  total  proportion  passing 
are  given) 


ITEM  GROUP 


B C D E 


# 

P 

•# 

P 

# 

p 

# 

p 

# 

P 

W-15 

.280 

0-63 

.445 

Op-54 

.615 

V-55 

.725 

W-l 

.865 

0p-60 

.325 

*-55 

.460 

Op-1 

.655 

E-ll 

.745 

0-53 

.900 

0-31 

*-39 

.360 

.380 

tf-16 

E-l 

.570 

.570 

h-46 

.705 

¥-18 

0-6 

.760 

,780 

0-8 

Op-2 

E-69 

„CXP> 

- ^ ^ 

.925 

.930 

CHAPTER  YI 


TgOOCOpB  TOR  TEE  CCKPj 
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of  hrotc  m oca  cmmemi 


Intrcdurtitn 

It  i ms  pointed  out  la  Chapter  I that  the  curricula,  for  HRUTC  and.  OCS 
candidates  differ  to  a considerable  «xtenfc--«eftainly  in  arrangement,  and 

to  EflWe  eoctent  la  slant  sad  nt*i+**rrr. . pnoywri  Ttyf.-t  rerr*  -f^rr  the  tWO  jcpQgapai 

most  of  necessity  he  different,  for  they  oust  reflect  the  curricular  differences. 
Therefore,  If  It  is  desired  to  ctnpare  graduates  of  the  two  programs  with  respect 
to  their  achlereMrjfft  in  naval  science,  it  is  lrgtossihle  to  use  the  examinations 
as  they  stand.  This  chapter  Is  concerned  slth.  an  attempt  to  adapt  the  examlna- 
tiaos  in  such,  a say  that  reascna&be  conparisons  between  HROTC  and  OCS  candidates 
efiuld  he  isade-  It  is  "believed  that  the  techniques  developed  here  could  also  he 
adapted  for  use  la  comparing  ROC  (Reserve  Officer  Candidate)  graduates  slth  the 
other  tso  groups. 

Hear  sell  do  4th  year  HROTO  students  retain  the  knosledge  acquired  during  the 
first  three  yearsT 

One  factor  which  militates  against  a valid  c caparison  of  HROTC  and  CCS 
graduates  1b  the  fact  that  in  HROTC  the  subject  •matter  is  Introduced  piecemeal 
over  a four -year  period,  whereas  in  OCS  six  different  (hut  somewhat  overlapping) 
subject -ant  ters  are  taught  together  in  the  course  of  in  weeks.  At  the  end  of  the 
OCS  course,  the  student  is  presumably  prepared  to  he  examined  on  any  subject  mat- 
ter, While  the  4th  year  HRCTC  graduate  Is  best  prepared  only  in  Engineering,  cer- 
tain aspects  of  Military  Justice , and  several  small  parts  of  other  subject- 


matters 


In  order  to  determine  hew  veil  fourth-year  HROTC  graduates  retain  the  material 
Of  the  first  three  years,  & special  5^-lten  exanri  nation,  hereafter  called  the  Cumu- 
lative ftomrl -nation,  v&a  comrtructed  for  administration  to  fourth-year  students  at 
certain  HROTC  units  in  May  1953*  This  examination  was  composed  chiefly  of  items  freon 
keys  El 00,  E200,  and  EjOO  of  the  OCS  examinations  vhich  appeared  to  be  appropriate  for 
testing  subject-matter  corered  in  the  HROTC  curriculum  of  the  first  three  years. 

Table  6.1  -identifies  the  items  On  this  examination.  The  items  were  classified  in  three 
keys,  FI,  F2,  and  F3>  corresponding  to  Key  ETOO,  E2.00,  and  E3Q0,  respectively. 

It  had  been  hoped  that  it  might  he  possible  to  administer  the  HROTC  Cumulative 
faaurrt  nation  to  & random  sample  of  HROTC  graduates,  using  various  HROTC  units  across  the 
country.  This  was  not  feasible,  however.  Circumstances  made  it  possible  to  give  the 
' examination  to  fourth-year  students  at  only  four  HROTC  units,  those  at  Harvard  Univer- 
sity (H  = 4l),  Yale  University  (R  = 50),  Brown  University  (H  - 36),  and  the  College  of 
the  Holy  Cross,  Worcester,  Massachusetts  (S  s 29),  a total  of  156  students.  Since  it 
was  thought  that  these  four  units  might  comprise  a biased  sample  of  HROTC  students, 
scores  an  the  OCB  were  obtained  for  the  students  tested,  as  a means  of  statistical  con- 
trol on  their  ability  level. 

Hext,  the  test  papers  of  the  item-analysis  sample  (H  = 200 ) of  OCS  Class  YIU  cases 
were  scared  with  keys  FI,  F2,  and  F3-  (it  will  be  recalled  that  these  scores  were  in- 
volved in  the  correlation  matrix  presented  in  Table  4.8. ) The  OCB  scores  were,  of  course 
alao  available  for  this  sample. 

Table  6.2  shows  the  means,  standard  deviations,  and  reliability  and  homogeneity 


coefficients  for  the  F-keys  of  the  cumulative  examination  in  the  OCS  sample  and  in  the 
HROTC  sample.  Two  major  conclusions  may  he  drawn  from  this  table.  First,  it  is  evident 
that  the  HROTC  group  did  less  well  than  the  OCa  group..  This  result  vac  expected,  because 
the  HROTC  Students  were  tested  on  materials  they  had  not-  reviewed  far  anywhere  from  one 
to  three  years.  Secondly,  the  measured  reliabilities  of  the  E-keys  were  much  lower  in 
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the  HROTC  group;  indeed,  the  reliability  of  Key  F-3  in  the  HROTC  group  is  probably  root 
significantly  different  from  zero.  This  result  is  probably  due  to  the  fact  that  the 
lack  of  review  caused  a marked  reduction  in  the  inter -correlations  of  the  items.  For 
this  group,  the  items  may  be  said  to  have  been  less  homogeneous- -which  items  the  individ- 
ual passed  was  more  a matter  of  chance.  In  cither  words,  the  items  were  not  remembered 
together;  the  loss  was  greater  for  some  items  than  for  others.  In  contrast,  the  003 
group  had  learned  and  reviewed  the  items  together;  they  had  higher  intercorrelations 
hence  higher  reliabilities.  These  results  are  a good  illustration  of  the  effects  of 
learning  and  forgetting  upon  test  reliability. 

It  may  also  be  noted  that  a possible  reason  for  low  reliabilities  In  the  HROTC 
group  may  have  been  the  inappropriateness  of  some  of  the  items  for  the  NF.QTC  curriculim. 
Perhaps  some  of  the  items  covered  material  which  had  never  been  naught  in  the  HROTC 
courses.  This  possibility,  however,  is  very  remote,  because  the  items  had  been  judged 
by  HROTC  instructors  as  appropriate,  and  several  students  volunteered  the  opinion  that 
the  Cumulative  Examination  was  perfectly  fair— the  fault  was  their  own  for  not  having 
remembered  the  material. 

Since  there  was  a possibility  that  the  two  groups  were  of  different  ability 
levels,  the  analysis  of  covariance  technique  was  used  to  control  any  such  differences 
statistically.  Actually,  the  NROTC  group  was  uniformly  superior  to  the  0C3  group  or. 

OCB  subtest  scares,  as  may  be  seen  from  Table  6.3;  therefore,  the  analysis  of  covariance 
was  hardly  necessary.  Nevertheless,  the  results  shown  in  Table  6.4,  which  presents 
covariance  analyses  for  the  three  F-scores,  will  remove  all  doubt  that  the  differences 
were  significant  and  in  a direction  favoring  the  OCB  group,  as  far  as  the  Cumulative 
Examination  is  concerned. 

Table  6.5  presents  the  intercorrelation  matrix  cf  F-scores  and  OCB  Sub-best. 

Scores  for  the  HROTC  group.  These  data  were  utilized  in  conjunction  with  data  shown 
in  Table  4.8  and  elsewhere  in  order  to  perform  the  analysis -of -covariance  computations 
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TABLE  6.2 

vmm,  standard  detuetcks,  and  reliability  and  HCMoggHgirr 

DATA  PCR  THE  7-ZECB,  CUMULATIVE  EXAMINATION 


OCS  Sample:  200  cues  in  Class  VIII  used  for  itera- 

tive factor -analysis  of  examinations. 

HROTC  Sample:  156  cases  at  Harvard,  Tale,  Brown  and 

Holy  Cross  administered,  the  Cumulative 
Examination. 


K - R 


Key 

SAMPLE 

N 

(Rights) 

A 

cr 

(20) 

rel. 

rel  (l  item) 

st 

FI 

19 

OCS 

200 

14.52 

2.0590 

.4197 

.0367 

.0768 

FI 

19 

HROTC 

156 

12.19 

2.1177 

.2779 

.0198 

.0381 

F2 

17 

OCS 

200 

12,81 

2.2827 

.5110 

.0579 

.0956 

F2 

17 

HROTC 

156 

10.47 

2.0707 

.2488 

.0191 

.0314 

n 

18 

OCS 

200 

12.32 

2.5725 

.5540 

.0646 

.n4i 

18  . 

HROTC 

156 

10.30 

1.9461 

.0690 

.oo4i 

.0070 

X 


TABLE  6.3 

COMPARATIVE  DATA  ON  OFFICER  CANDIDATE  BATTERY  SUBTEST  SCORES 

OCS  Sample:  200  cases  in  Class  VIII  used  for  Itera- 

tive factor -analysis  of  examinations. 

HROTC  Sample:  156  cases  at  Harvard,  Yale,  Brown  ard 

Holy  Cross  administered  the  Cumulative 
Examination 


OCB  Subtest 


OCS  Sample 
X 


HROTC  Samp] e 
X 


Verbal  Reasoning 

55.27 

8.4486 

62.22 

7.2176 

Mechanical  Comprehension 

53.42 

10.5092 

59.12 

8.8603 

Mathematical  Comprehension 

50.74 

8.5025 

59.28 

8.7057 

Relative  Motion 

45.88 

7.7183 

52.02 

9.6626 

Spatial  Relations 

Phi? 

9-3714 

57.37 

8.9164 
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presented  in  Table  6.k,  It  nay  be  noted  that  the  analysis  of  covariance  performed 
here  was  somewhat  unusual  in  that  it  utilised  five  control  variables  (i.e.  the  five 
CCB  Subteat  Scores).  Special  matrix  copgutational  procedures  ware  devised  in  order 
to  accomplish  this. 


L»lr 


of  OCS  and  HROTC  students  oh  items 


32 


reviewed 


It  was  established  in  the  preceding  paragraphs  that  4th  year  HROTC  students 
do  not  retain  the  material  of  the  first  three  years  very  well.  True,  some  of  them 
do,  but  an  the  average  they  do  not.  The  question  may  be  asked,  however,  what  the 
results  would  be  if  OCS  students  are  compared  with  HROTC  students  on  material  which 
they  have  both  freshly  reviewed.  That  is,  hcrw  do  OCS  students  compare  with  HROTC 
students  of  the  4th  year  on  material  covered  in  the  fourth  year  of  the  KROTC  cur- 
riculum? How  do  OCS  students  compare  with  HROTC  students  on  material  covered  in 
the  third  year,  and  so  ant 

The  procedure  used  in  answering  these  questions  was  as  follows: 

(1)  All  items  common  to  the  May,  1952  HROTC  examinations  (by  year)  and  the 
December  1952  OCS  examinations  (by  department)  were  identified.  Small  variations 
in  items,  such  as  an  editorial  change  in  the  lead,  and  even  a few  major  variations 
such  as  the  substitution  of  an  alternative,  were  ignored  in  identifying  common  items 
The  items  were  classified  in  terms  of  whether  they  appeared  on  eny  of  the  E-keys 
(E1Q0,  E200,  F300  and  E400).  The  results  of  this  analysis  are  shown  in  Table  6.6. 

(2)  It  had  been  hoped  that  the  conparison  could  be  made  separately  for  each 
of  the  E-keys,  and  for  each  of  the  four  years.  This  proved  impossible  in  view  of 
the  fact  that  not  enough  common  items  were  available  in  each  E-key  id  in  each  year. 
It  would  have  been  possible,  of  course,  to  use  all  items  common  to  the  Examinations, 
regardless  of  their  appearance  in  E-keys,  but  the  resulting  sets  of  items  would  have 
been  heterogenous,  and  many  of  them  would  be  of  insufficient  validity. 


T9 


1A3LS  6.4 

COTiBIiXCX  /OCALTHEB  FX  TERSE  F-KET  SCOCB3  TBEBEHg  TEE  SiaXFTCjQICI 

dr  tee  suraoBRSE  bchukkk  ocs  akd  more  oroqps  jtthi 


Aorasnoarr  is  mates  r®  ocs  ixnTSREiCBa 


TOTAL 


2027.00  .9516  1929.50  350 


TOTAL 


2190.89  .9586  2100.19  350 


TOTAL  2271.49  .9272  2106.13  350 


significant  far  beyond  the  . 001  Lerel. 
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} Adjusted  d.f. 

Mean 

7 

S.S. 

Square 

1 

rWAtrr-p  £ ^ c 

INTERCCKRELATI0N8  Or  THE  OFFICER  CANDIDATE  BATTERY  AND  "F1  SCORES 
NROTC  CumLative  Examination  Sample  (N  « 156  students  at 

Harvard,  Holy  Cross,  Brown,  Yale) 


• 

Variable 

1 

2 

3 

4 ' 

5 

6 

. 7. 

8 

1- 

OCB  (Verbal  Reasoning) 

1.00 

.388 

.338 

.152 

.264 

.381 

.315 

.345 

2 

OCB  (Mechanical  Comp. ) 

.398 

1.00 

.410 

.322 

.500 

.213 

•339 

.246 

3 

OCB  (Mathematical  Comp.  ) 

.338 

.410 

1.00 

.275 

.442 

.126 

.237 

.154 

4 

OCB  (Relative  Motion) 

.152 

.322 

•375 

1.00 

.352 

.086 

.270 

.146 

5- 

OCB  (Spatial  Relations) 

.26'; 

.500 

.442 

.352 

1.00 

.112 

.227 

.175 

6 

F 1 Key 

** 

U) 

CD 

H 

.213 

.126 

.086 

.112 

1.00 

.278 

.207 

7 

F 2 Key 

•315 

.339 

..237 

.270 

.227 

.278 

1.00 

.337 

(T 

F 3 Key 

.3^5 

.246 

.154 

.146 

.175 

.207 

.337 

1.00 
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TABLE  6.6 

ITE36  WHICH  ARE  FOUND  IN  BOTH  THIS  DEC.  1952  OCS  SUO G AND  IN  MAX  1952  NROTC  EXAJB 
WITH  THE  CORRESPONDING  ITEM-NUMBERS  IN  EACH  EXAMINATION 


INCUJDED  IN  E KEYS 

T!vT«>iM 

i&p 

0P46 

lib 

0 15 

T 

0P51 

117 

0 23 

39 

OP35 

11.4 

0 68 

77 

OP39 

119 

32 

88 

QP4l 

11.8 

33 

92 

o'29 

84 

335 

130 

0 10 

82 

S21 

137 

0 14' 

75 

328 

135 

0 7 

5 

S25 

139 

I*  I 


OP69  111 

W47 

55 

• W4l  -• 

148 

W15 

0P71  1 12 

W58 

63 

W43 

119 

W48 

W69 

: 77 

W52 

1 61 

W72 

8 ? 

OP  58 

124 

QP68 

113 

w45 

•-  2 

3NS-1  N17 


0P30  20 

6 


41 

E36 

5 

7 E5 

13 

E45 

io4 

0 75 

142 

E15 

11 

E4 

•16 

E7 

25 

0 52 

123 

0 65 

135 

El4 

14 

E19 

40 

El6 

31 

0 54 

127 

E51 

93 

e6 

20 

0 70 

146 

' • E12 

34 

0 71 

148 

E13 

2? 

0 (2 

147 

E9 

36 

0 73 

150 

E18 

45 

E17 

39 

E22 

44 

E23 

47 

E50 

94' 

E34 

55 

B53 

97 

E 66 

64 

• ■ 

■ •"  - : 

■ E55 

99 

■■■  “•  "i 

E?2 

67 

0 46 

124 

E74 

72 

- 

0 53 

117 

E75 

79 

E 24 

51 

E70 

61 

... 

. . . "1. 

E28 

53 

E43 

86 

E37 

60 

e42 

106 

E68 

e44 

65 

8? 

e48 

S46 

89 

88 

GtesLenfca&ion  E « Engineering  S = S^eucansb 
Operations  N = navigation.  W = Weapons 


It  was  therefore  decided  to  make  cooperative  studies  only  for  the  second,  third, 
and  fourth  years  of  KROTC  (i.c., occluding  the  first  year),  and  to  use  only  items  ap- 
pearing on  key  3200  and  key  3300.  The  item  frra  these  two  keys  were  ccstbined  <m  the 
ground  that  they  were  essentially  equlyalest,  since  keys  3200  and  3300  had  previously 
been  shown  to  be  highly  correlated  (see  Chapter  IV).  Thus,  for  comparative  purposes 
three  sets  of  item  were  established,  identified  as  Key  323-2,  523-3,  and  323-4,  re- 
spectively, far  the  second,  third,  and  fourth  years. 

(3)  These  keys  were  applied: 

(a)  To  4 00  cases  from  OCS  Class  7XH  (i.e.,  the  factor-analysis  and  the 
cross-validation  samples  combined),  and 

(b)  to  representative  samples  of  KROTC  students  tested  in  May  1952*  Four 
hundred  cases  in  each  year  were  studied.  These  samples  of  paper?  had 
been  selected  by  the  Bureau  of  Koval  Personnel  as  representative  of 
the  several  thousand  cases  tested  at  all  52  KROTC  units. 

The  statistical  results  are  shown  in  Table  6.T.  Since  OCB  scares  were  not  readily 
available  for  the  NR0TC  examinees,  it  was  inpossible  to  adjust  (by  the  analysis  of  co- 
variance  technique)  for  any  differences  in  the  ability  level  of  the  respective  groups. 
Nevertheless,  it  should  be  pointed  out  that  there  is  little  reason  to  fear  serious 
bias  in  the  samples,  since  each  was  a random  sample  from  its  respective  group. 

For  the  4th  and  3rd  year  comparisons,  the  differences  in  means  are  significant 
far  beyond  the  .001  level,  and  favor  the  NROTC  group.  The  difference  in  the  2nd -year 
comparison  also  favors  the  NROTC  group,  but  was  not  significant  at  the  .05  level.  It 
ia  impossible  to  assign  causes  for  these  differences,  in  the  absence  of  other  informa- 
tion. The  differences  may  be  due  to  any  one,  or  several,  of  the  following  conceivable 
conditions:  (a)  the  KROTC  group  my  be  of  a higher  average  ability  level;  (b)  the 

KROTC  group  may  be  able  to  learn  better  because  of  longer  periods  of  time  available. 

In  contrast  to  the  l£ -weeks  course  at  OCS  where  all  subjects  are  presented,  in  sixteen 
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TABLE  6.7 


Year  Group 


*P<  .001 


CCKt'ARISCN  OF  OCS  AND  NROTC  GROUPS  QN  X23-KEIS 
N = 1+00  for  each  group 


(Rights)  a 


(20) 

rel. 


rel. 

(l  item) 


4 

NROTC 

OCS 

B23-4 
(22  items) 

15.99 

14.24 

3.23 

3«i4 

7.78* 

.650 

.585 

kk 

O 03 

.116 

.095 

3 

NROTC 

OCS 

123-3 

(15  items) 

IO.38 

9.18 

2.42 

2.42 

7.03* 

.612 

.624 

.095 

.100 

.184 

.214 

2 

NROTC 

OCS 

E23-2 
(12  items) 

8.86 

8.64 

1.98 

1.86 

1.64 

.505 

.390 

.078 

.051 

.118 

.082 
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By  caoqputaticm  from  the  frequency  distributions  of  scopes,  also  given  in  the  table, 
it  would  then  appear  that  (at  least  In  the  staples  used  for  this  analysis)  31^,  or 
78.5^  of  the  HROTC  group  attained  the  critical  score  of  14,  while  only  23$,  or 
59«7 5^  of  the  OCS  group  attained  this  score. 

Sunaary 

From  the  standpoint  of  test  results  from  representative  samples,  it  would 
seem  that  the  HROTC  graduate  is  a slightly  better  product,  on  the  average,  than 
the  OCS  graduate,  at  least  when  he  is  ccnpared  with  respect  to  material  he  has 
freshly  reviewed.  The  fourth-year  KROTC  student  does  nrrt,  however,  retail  the 
material  presented  In  the  first  three  years  as  well  as  might  be  desired.  On  the 
other  hand,  he  may  he  in  a better  position  to  relearn  it  when  this  knowledge 
becomes  of  use  to  him  in  his  duties  in  the  fleet. 


r ; .1  :'ii  . •» .. 
.0  ‘ • •.  *.  rf .-Jr* 

*•;  i "...ih*  •: 


. - oi'suBriKQ'  cwRAcsauEEXo  am.  ■ 

. Key t.v'  v . • 

800  (400  Sad  year  KROTC  cues  tested  May,  1952; 
400  DCS  cases  tested  Bee.-:,.  1952)  1 


Item 

throop 

: •• 

Proportion  Passing 

Normal 

deviate  3~5 

’"^valu^'  ‘4.5 

Score.  Group*  (Range  end  Kean)  '(Rights) 

6 : ' t :s  " 8 w ''9  10  11 

12 

Avi  *• 

.299 

*371 

’ *491 

• .517.'  iv 

1624 

-.725'  •• 

.842 

1.000 

B 

-,65 

.366 

.457 

.587 

.662 

.759 

.880 

.939 

1.000 

c . 

~.84 

. .484-. 

• 59T- 

•.  .625  - 

»•  ,.f( 4 .,;- 

-.828  • 

.876 

• .95^ 

1.000 

D 

-99 

.336 

.677 

.700 

.812 

.377 

.913 

.974 

1.000 

‘ •• 

Cases  in  KROTC 

23 

. 33. 

..  36 

64  ... 

79 

78 

56 

31 

scdtfe  grddpj  - t>CS  r' 

28'  ' 

29 

• -1^ 

69 

"80 

94 

«■.  ; 

42 

l4 

TOTAL 

51 

62 

80 

133 

159 

172 

98 

45 

SPECIFICATION  OF  ITEM  GROUPS* 


ITEM.  GROUP 

A B C D 


2NSjf 

PN 

PO 

2NS# 

PN 

PO 

2NS# 

PN 

PO 

2NS# 

PN 

PO 

65 

.595 

.620 

61 

.745 

.762 

148 

.720 

.868 

H9 

.875 

.805 

113 

.845 

.200 

1 109 

.735 

.765 

2 

.733 

.852 

l40 

.915 

.723 

55 

.420 

.610 

124 

.840 

.585 

77 

.795 

.797 

85 

.635 

.330 

* For  each  item,  its  number  in  the  2NS  examination  is  given,  as  veil  as  its  difficulty 
Index  in  the  KROTC  group  (PN)  and  in  the  OCS  group  (PO). 


.y . 1 
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TABLE  6.9 

C3PERA2HKG  CHARACTERISTIC  BATA. 


Key  E23-3 

If  = 800  (400  3rd  year  HROTC  cases  tested  May,  1952; 
400  OCS  cases  tested  Dec.,  1952) 


A +.35  .071  .121  .209  .266  . 342  A52  .542  . 700  .894 
B ...29  .220  .382  .435  . 540  .646  .718  .817  .882  .957 
C -.65  .381  .539  .827  .710  .784  .807  .870  .909  .968 
D -l.,69  .771  .845  . 900  .923  .934  . 977  . 988  .982  .995 


Ho.  of  cases  J HROTC  26  20  36  50  69  60  62  4l  36 

in  score  ) OCS  5 8 48  56  57  56  49  46  14  16 


SPECIFICATION  OF  ITEM  GROUPS* 


29 

.505 

.418 

34 

.693 

.650 

26 

.870 

.677 

22 

.983 

.982 

28 

.423 

.487 

2 

.710 

.550 

l 

.740 

.707 

4 

.963 

.970 

30 

.330 

.240 

31 

=733 

.502 

33 

.507 

.868 

32 

.947 

.928 

24 

• 377 

.195 

23 

.813 

.227 

12 

.850 

.932 

*For  each  item,  its  number  in  the  3-NS  examination  Is  given,  as  well  as  its 
difficulty  index  in  the  HROTC  group  (p^)  and  in  the  OCS  group  (p()). 


i 


i 

l 

a 

£ 

s 
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TABLE  6.10 

OPERATING  CHARACTERISTIC  DATA 
Kay  E23-4 

800  (400  4th  year  WROTC  esses  tested  May,  1952? 
1*00  OCS  cases  tested  Dec.  1952) 


1cm  Pass 


normal 

So 

ore  Gr< 

sups  t 

;rrr,!f 

and  Meu 

#nj  (? 

Rights 

r~ 

deviate 

5-olo 

11-12 

21*22 

ESI 

8:9 

11.6 

13 

14 

. 15 

16 

17 

18 

1? 

20 

£1-4 

+.18 

.151 

.250 

.312 

.335 

.424 

.443 

.569 

.593 

.671 

.803 

.940 

-.34 

.286 

.412 

.488 

.531 

.618 

.724 

.727 

.781 

.876 

.913 

.983 

w«59 

.*33 

.542 

.647 

.710 

.719 

.742 

.818 

.866 

.881 

.924 

.959 

-.92 

.570 

.688 

.733 

.794 

.833 

.854 

.873 

.936 

.949 

.946 

.974 

-I.28 

.714 

.859 

.884 

.910 

.889 

.920 

.926 

.960 

.951 

.976  : 

1.000 

(NROTC  25 

31 

30  : 

J6  38 

48 

62 

33 

4o 

33  : 

t oca  53 

66 

42  1 

*■9  34 

48 

46 

34 

14 

9 

TOTAL  78  97  72  85  72  96  108  67  51*  42  29 


SPECIFICATION  OF  ITEM  GROUPS* 


ITEM  GROUP 
C 


*For  each  Item,  its  member  in  the  4-NS  examination  is  given,  as  well  as  its 
difficulty  index  in  the  NROTC  group  (py)  and  in  the  OCS  group  (Pq)» 
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CHAPTER  VI T 


SUMMARY  AND  CONCLUSIONS 


This  project  had  as  its  objective  uhs  construction  of  a new  series  of  exami- 
nations for  measuring  the  "integrated"  knowledge  about  naval  science  demonstrated 
by  graduates  of  naval  officer  candidate  courses. 

The  definition  and  description  of  what  "integrated"  knowledge  would  consist  of, 
and  how  it  could  be  measured,  constituted  the  major  technical  problem  arising  in 
this  research.  Two  approaches  to  this  problem  were  explored; 

1.  An  attempt  was  made  to  define  "integrated"  knowledge  by  logical  and 
rational  analysis. 

2.  An  attempt  was  made,  by  means  of  special  statistical  analyses  of  test 
results,  to  isolate  several  linearly  independent  factors  of  achievement. 

It  was  hoped  that  one  or  more  of  these  factors  would  appear  to  correspond 
to  the  notion  of  "integrated  knowledge"  developed  in  the  rational  analysis 
of  this  problem,  and  that  these  factors  would  appear  to  be  distinguish- 
able from  other  factors  which  in  turn  would  correspond  to  the  notion  of 
"non-integrated,"  "mere  factual"  knowledge. 

In  the  course  of  the  project,  various  new  examination  materials  (items)  were 
developed,  both  for  use  at  the  Officer  Candidate  School  (Nevmcrt,  R»  I.)  and  a1 
the  $2  NROTC  units  throughout  the  country.  The  logical  analysis  of  the  problem  of 
“ integrated"  knowledge  was  used  as  a guide  in  constructing  these  examinations,  and 
samples  of  completed  answer  sheets  resulting  from  the  administration  of  the  exam- 
inations were  used  as  the  raw  data  for  the  statistical  analysis. 

Logical  analysis  appeared  to  disclose  three  possible  ways  of  defining 
"integrated  knowledge,"  at  least  in  terms  of  the  types  of  test  items  which  might 
be  used  in  testing  such  knowledge:  (a)  items  testing  the  ability  to  utilize  facts 
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from  different  branches  of  subject-matter)  (b)  items  depicting  realistic  situations 
drawn  from  job-analyses  of  shipboard  duties  of  junior  naval  officers)  and  (c)  items 
which  would  require  not  only  the  knowledge  of  relevant  facts  but  also  the  ability 
to  weigh  various  elements  in  the  situation,  thus  testing  decision-making  and 
problem-solving  functions.  To  some  extent,  the  definitions  are  not  mutually  ex- 
clusive, and  all  three  definitions  were  used  in  constructing  "integrated"  items, 
although  definition  (c)  was  accepted  as  most  generally  useful  and  practicable. 

Attempts  were  therefore  made  to  construct  items  which  presented  realistic  and 
relatively  complex  problem-situations  involving  a number  of  basic  knowledges  and 
requiring  the  evaluation  of  various  elements  in  these  situations.  One  particularly 
promising  technique  was  the  construction  of  series  of  situationally  related  items) 
in  each  series,  some  items  could  not  be  answered  by  themselves  but  required  infor- 
mation presented  cr  implied  in  previous  items.  It  was  hoped  that  such  series  of 
items  would  measure  the  ability  to  perceive  the  various  elements  involved  in  a 
situation.  Such  an  ability  would  presumably  be  still  another  facet  of  "integrated 
knowledge . " 

In  practice,  it  was  found  that  certain  branches  of  subject,  matter  lent  themselves 


to  the  cont -.ruction  of  "integrated"  items  better  than  other  branches.  For  example, 
the  subject  matter  known  as  Naval  Operations  lends  iteself  to  items  of  the  problem- 
solving and  decision-making  type  better  than  the  subject  matter  of  Naval  History,  to 
take  a rather  obvious  example.  This  is  not  to  deny,  of  course,  that  such  a subject 
matter  as  Naval  History  can  be  tested  from  the  standpoint  of  "integrated"  knowledge 
and  unaerstrnding,  but  it  did  not  lend  itself  to  the  approach  adopted  in  the  present- 
research. 

The  success  of  the  project  in  constructing  "integrated  examinations"  is  chiefly 
a matter  of  subjective  opinion,  since  there  was  not  objective  criterion  av  axlable  for 


ascertaining  the  extent  of  this  success.  In  the  last  analysis,  the  results  can  be 
judged  only  by  a subjective  evaluation  of  the  examinations  themselves. 


The  statistical  analysis  of  te3t  results  attempted,  however;  to  see  whether 
different  varieties  of  achievement  could  be  identified.  The  Wherry-Gayloid  iterative 
factor -analysis  procedure  was  applied  to  tie  examinations  administered  to  the 
December,  1952  graduating  class  at  the  Officer  Candidate  School.  Of  the  four 
interpretable  dimensions  of  achievement  identified  in  this  manner,  two  were  sc 
highly  correlated  and  so  nearly  alike  in  appearance  that  they  were  considered  as 
one.  This  one  dimension  contained  the  largest  number  of  items  and  was  interpreted 
as  "achievement  in  problems  requiring  spatio-temporal  reasoning,"  It  was  also 
the  dimension  best  predicted  by  the  subtests  of  the  Officer  Candidate  Battery. 

The  other  two  dimensions  were  tentatively  interpreted,  respectively,  as  "achievement 
in  problem  requiring  verbal  reasoning  and  judgment"  and  "achievement  in  tesu  items 
requiring  chiefly  recall  of  isolated  facts. " Thus,  at  least  two  linearly  in- 
dependent dimensions  of  achievement  seem  to  correspond  to  the  definition  of  an 
"integrated"  examination  as  one  involving  problem-solving  and  decision-making,  and 
a third  dimension  of  achievement  seems  to  correspond  to  the  notion  of  a "non- 
integrated"  examination  as  one  which  merely  involves  the  recall  of  isolated  facts. 

It  should  be  said,  however,  that  these  dimensions  seem  to  be  rather  highly  inter- 
correlated,  even  though  presumably  linearly  independent  in  the  sense  customarily 
employed  in  factor -analytic  theory.  That  is,  officer  candidai.es  tend  to  perform  at 
about  the  same  level  on  all  three  dimensions  of  achievement.  It  is  entirely 
possible  that  the  three  dimensions  isolated  arose  chiefly  through  errors  of  sampling 
this  possibility  could  be  checked  in  future  investigation.  Furthermore,  the 
subjective  interpretations  which  have  been  made  for  the  factors  are  possibly  open 
to  argument.  In  any  case,  .0  was  concluded  that  there  is  a very  strong  "general" 
factor  of  achievement  in  officer  candidate  programs.  In  other  words,  there  is 
one  major  dimension  of  achievement,  and  this  dimension  is  measured  almost  equally 
well  by  "integrated"  items  of  the  type  specially  constructed  in  this  project 
and  by  "non-integrated"  items  which  chiefly  demand  recall  of  factual  information. 
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This  is  not  to  say  that  the  effort  to  construct  examinations  of  an  "integrated" 
type  (in  contrast  to  "non-integrated"  examinations)  is  futile . Or.  the  contrary, 
it  is  probably  desirable  to  emphasize  problem-solving  and  decision-making  in  the 
items  of  an  examination  battery  for  officer  candidates,  even  though  students  perform 
about  equally  well  on  either  type.  Educators  have  consistently  held  to  the  view  that 
examinations  should  reflect  the  objecti  es  of  the  curriculum.  It  is  believed  that 
naval  training  authorities  intend  that  their  officer  candidate  curricula  seek  (among 
other  things)  to  teach  the  ability  to  solve  the  complex  problems  arising  in  fleet 
operations  and  tc  make  the  sound  decisions  called  for  in  these  operations.  This 
being  so,  naval  science  examinations  should  as  far  as  possible  present  problems  and 
situations  calling  for  decision-making  similar  to  those  which  might  be  encountered 
in  fleet  operations. 

Educators  have  also  had  the  experience  that  examinations  set  by  external  or 
higher  authority  tend  to  have  an  influence  on  what  is  emphasized  in  a curriculum 
and  how  it  is  taught.  This  influence  can  be  made  beneficial  if  the  examinations 
reflect  the  intentions  of  these  authorities.  Thus,  there  is  every  reason  to 
recommend  that  continued  efforts  be  made  to  assure  that  the  examinations  prepared  for 
naval  officer  candidate  programs  emphasize  those  types  of  achievement  believed  most 
desirable,  regardless  of  the  fact  that  students  appear  to  perform  at  about  the  same 
level  on  various  types  of  examination  materials. 

One  of  the  relatively  novel  features  of  the  present  research  was  the  application 
of  a test-analysis  technique  called  the  "technique  of  individual  operating  character- 
istics." This  technique  is  designed  to  disclose  the  meaning  of  a test  score  in 
terms  of  the  level  of  performance  which  it  implies  on  test  items.  Thus,  instead 
of  interpreting  a test  score  in  terms  of  its  relative  position  in  a group  of  scores 
(as  is  done  with  the  usual  percentile  and  standard  score  techniques),  one  is  enabled 
to  interpret  it  in  terms  of  what  item  difficulty  level  the  individual  is  able  to 
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master  with  some  specified  probability.  By  the  inspection  of  items  at  this 
difficulty  level,  one  may  therefore  judge  the  satisfactoriness  of  the  performance 
represented  by  a given  score.  It  is  believed  that  the  data  assembled  by  means  of 
this  technique  and  presented  in  this  report  will  enable  naval  training  authorities 
to  evaluate  the  achievement  of  naval  officer  candidates  more  realistically  than 
could  be  done  by  means  of  previously  available  techniques. 

A final  problem  to  which  this  research  sought  answers  was  th  > comparison  of 
officer  candidates  in  the  several  officer  candidate  programs.  In  »his  research, 
data  were  accumulated  and  analyzed  in  such  a way  as  to  display  the  differences,  if 
any,  between  graduates  of  the  NROTC  program  and  the  OCS  curriculum  at  Newport,  R.I. 
The  analysis  was  based  upon  those  examinati  Cn  CCTT.TiOT"?  t/O  t-ho  two  curricula  and 

to  the  corresponding  examinations.  These  comparisons  had  to  be  made  separately 
for  each  year  of  the  NROTC  curriculum,  which,  of  course,  is  spread  out  over  a 
, four-year  period  rather  than  being  concentrated  in  an  18-week  course  as  it  is  at 

the  Officer  Candidate  School,  In  this  series  of  comparisons,  based  on  rep- 

* 

res&ntative  samples  of  NROTC  and  OCS  students,  the  average  NROTC  student  was 
consistently  superior  to  the  average  OCS  student,  (There  was,  of  course,  consider- 
able overlap  in  the  distributions,  however.)  The  superiority  of  the  average  NROTC 
student,  was  increasingly  evident  from  the  second  to  the  fourth  year  of  the 
NROTC  curriculum^  the  most  probable  explanation  of  this  resu.lt  is  that  NROTC 
students  are  increasingly  highly  selected  as  groups  pass  through  the  four  years  of 
NROTC  program. 

These  comparisons  of  NROTC  and  OCS  students  were  based  on  examination  materials 
freshly  reviewed  by  each  group j for  example,  OCS  students  were  compared  with 

» 

fourth-year  NROTC  students  only  on  examination  materials  found  in  the  fourth-year 
» NROTC  curriculum.  It  was  of  interest,  however,  to  ascertain  the  extent  to  which 

fourth -year  students  retain  the  material  learned  in  the  earlier  years  of  the 
curriculum.  For  this  purpose,  a special  examination  covering  first-,  second-,  and 
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third-year  NROTC  material  also  appearing  in  the  OCS  Curriculum,  was  prepared  and 
administered  to  a selected  sample  of  fourth-year  NBOTC  students  in  four  units  in 
the  New  England  area*  The  scores  of  these  students  were  compared  with  the  scores  of 
OCS  students  on  the  same  items j scores  on  the  Officer  Candidate  Battery  were  used 
as  a oasis  for  statistical  control  of  ability  level.  Use  of  the  analysis  of 
covariance  technique  revealed  that  the  OCS  students  were  very  significantly 
superior  to  the  NROTC  students  on  this  special  examination.  This  result,  of  course, 
was  not  unexpected,  since  the  OCS  students  had  freshly  reviewed  the  material  while 
the  NROTC  students  had  not.  It  does  point  up  the  fact,  however,  that  NROTC  students 
in  their  fourth  yeqr  do  not  retain  the  material  taught  in  earlier  years  to  the 
extent  that  might  be  desired.  In  view  of  the  fact  that  there  is  no  formal  provision 
for  cumsilative  review  in  the  NROTC  curriculum,  naval  authorities  may  wish  to  consider 
tne  implications  of  these  results  for  a possible  modification  in  the  plan  of  the 
NROTC  curriculum. 

As  a final  word  of  summary,  it  may  be  stated  that  the  examinations  which  have 
been  in  use  in  the  NROTC  and  OCS  programs  of  the  past  several  years  are,  in  the 
judgment  of  the  present  investigators,  reasonably  reliable  and  valid  examinations. 
They  have  been  well  planned,  they  represent  adequate  samplings  from  the  respective 
curricula,  and  they  function  well  in  discriminating  officer  candidates  of  high  and 
low  achievement.  The  present  research  has  explored  some  ways  in  vrhich  the 
examinations  could  be  further  improved,  and  some  progress  in  actually  improving 
the  examinations  may  be  claimed. 

Several  recommendations  for  further  research  may  be  made.  First,  in  the 
construction  of  new  examination  materials,  the  approach  utilized  in  the  present 
project  could  be  further  explored.  Items  identified  in  this  report  as  appearing  to 
measure  problem-solving  and  decision-making  ability  couid  be  used  as  models  for  the 
construction  of  new  examination  items.  This  approach  could  be  extended  into  certain 
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areas  of  the  naval  science  curriculum  which  were  not  thoroughly  worked  on  in  the 
present  project,  such  as  Engineering  and  Damage  Control,  and  Military  Justice. 
Descriptions  of  realistic  situations  which  would  provide  the  necessary  context 
for  the  construction  of  " integrated"  items  could  be  further  developed.  The 
preparation  of  series  of  inter-related  items  based  on  a single  situation  appears  to 
be  a particularly  promising  technique. 

In  the  meantime,  the  special  factor  keys  developed  in  the  present  research 
could  be  validated  against  on-the-job  shipboard  performance.  This  would  require 
tracing  the  present  assignments  of  former  graduates.  The  factor  keys  could  also 
be  used  as  tentative  criteria  for  new  items  which  might  be  constructed^  if  this 
plan  were  to  be  followed,  the  items  composing  these  factor  keys  should  continue 
to  be  included,  in  future  examinations.  Concurrently,  further  research  could  be 
undertaken  to  examine  the  factorial  composition  of  naval  science  examinations  in 
order  to  see  ’.Aether  further  evidence  could  be  found  for  differentiating  factual- 
knowledge  examinations  from  what  have  been  described  as  "integrated"  examinations 
in  the  present  report. 


